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ABSTRACT 


This thesis developed behaviorally anchored-rating scales (BARS) for use in evaluating 
Marine Corps small-units (SUs) during live and virtual decision-making (DM) training. 
Currently, the Marine Corps does not mandate the use of standardized metrics and 
processes for the evaluation and feedback during SU DM training scenarios. Often, 
evaluators assess trainee performance in uncertain situations based on gross outcomes 
rather than process-oriented measures. Properly developed and integrated into a training 
plan, BARS offers a novel solution to these constraints. The application of BARS has 
been explored in the law enforcement and medical domains and has been explored on a 
limited basis for infantry tasks. This effort proposed that observing SU leader’s 
observable behaviors in context of their task performance would provide objective 
measures of DM. The first portion of this work was focused on the design and 
development of BARS specific for use during multiple IIT scenarios. The second phase 
of this research refined the initial BARS into six scenario-specific evaluation 
measurement tools. Additionally, this work provided an initial effectiveness evaluation of 
the refined BARS in operational training at the IIT. Results indicate that the BARS 
developed for this thesis have the potential to replace the nonstandard and subjective 
methods currently used for the evaluation of DM training in live and virtual 
environments. 
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I. 


INTRODUCTION 


The United States Marine Corps (USMC) has been fighting and defeating our 
nation’s enemies for 237 years. Regardless of the clime or place, the Marine Air-Ground 
Task Force (MAGTF) places ever-increasing responsibility on its Marines, particularly at 
the non-commissioned and company officer grades in order to remain adaptable to any 
environment (Hannigan et ah, 2012). The concept of the “Strategic Corporal,” a term 
coined by Gen Charles C. Krulak, emphasizes that actions made by individual Marines at 
the tactical level can influence the operational and strategic levels of warfare (Krulak, 
1999). This type of individual decision-making at the tactical level makes educating and 
training marines and sailors in ethics and decision-making in complex and uncertain 
environments imperative (Kobus, 2012). Concomitantly, a means to measure and 
evaluate a Marine’s or Sailor’s decision-making and leadership ability in such 
environments is critical to enabling the future success of the MAGTF (Dunford, 2011). 

The ambiguous and complex nature of modern warfare puts a premium on small- 
unit leadership at the squad, platoon, and company organizational levels (Gideons, 2011). 
Leaders reacting to a dynamic and chaotic battlespace, which is compounded by 
geographic dispersion, mission diversity, adaptive adversaries, and rules of engagement 
(ROE) (Board, 2012), are required to make intelligent, timely, and situationally relevant 
decisions. Additionally, deficiencies in time and information further complicate a 
leader’s ability to fully process and analyze the myriad of possibilities. 

On an important decision one rarely has 100% of the information needed 
for a good decision no matter how much one spends or how long one 
waits. And, if one waits too long, he has a different problem and has to 
start all over. This is the terrible dilemma of the hesitant decision maker. 
(Greenleaf & Spears, 2002, p. 36) 

Marines and sailors need to be able to quickly and accurately make decisions in 
combat, despite the complexities, and instructors and supervising officers need tools to 
help them evaluate the trainees’ ability to do so. The art and science of decision-making 
encompasses more than a decision maker’s speedy computational prowess. Decision¬ 
making requires situational awareness of the problem at hand, along with the creative 
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means, informed by experience, education and intelligence, to achieve the desired 
solution (USMC, 1997). 

Colonel John Boyd’s (USAF) “Observe, Orient, Decide and Act” (OODA) loop is 
an abstraction of the decision-making cycle that is reliant upon an individual first 
observing the environmental conditions or cues. Once the individual has detected the 
cues in the environment, the Marine or Sailor must adjust his or her mental picture to the 
situational reality and then decide on a course of action or inaction (Angerman, 2004). 
This decision-making cycle is a descriptive framework that military leaders and 
organizations use to understand and analyze the human command and control (C2) 
system (Hannigan et ah, 2012). 

Decision-making theories and models are categorized as involving either 
analytical or intuitive strategies (Sjoberg, 2003). An example of an analytical strategy is 
found in the Vroom-Jago “Rational Decision” model (Turpin & Marais, 2004). Gary 
Klein’s “Recognition Primed Decision” (RPD) model and Boyd’s “OODA” theory are 
categorized as intuitive strategy models (Turpin & Marais, 2004). Analytical decision 
models are more appropriate for simple scenarios where time is sufficient to evaluate 
alternatives for a relatively few courses of action (COAs) and the decision maker has 
access to the information needed to make a decision. Intuitive decision models are more 
appropriate for complex or uncertain scenarios where deficiencies in time, information 
and/or resources will not impede the selection of a sufficient CO A (Roth, 2004). 
Additionally, intuitive decision-making strategies take into consideration the operational 
factors experienced by deployed Marines and Sailors (Commandant of the Marine Corps, 
2010B). Due to these factors, this research will focus on the intuitive strategies instead of 
analytical methods when discussing the decision-making process. 

The Commandant of the Marine Corps (CMC), Gen. James F. Amos, established 

a set of priorities for the current and future outlook of the USMC. His third doctrinal 

priority is, paraphrasing, to improve training and experience levels for maneuver unit 

squad leaders (Amos, 2010). Additionally, the CMC specifically points out that the 

Naval Postgraduate School and marines attending the institution direct their individual 

studies to “ensure they align with USMC needs” (Amos, 2010, p. 14). Science and 
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Technology Objectives (STOs) codify these identified gaps and address them under the 
purview of the Office of Naval Research (ONR). Research conducted for this thesis will 
support the Human Performance, Training and Education (HPT&E) thrust, aligning with 
the Test and Evaluation (T&E) STO #2, “Small unit learning and performance 
assessment” (ONR, 2012). 

Current operational training of Marine Company, Platoon, and Squad-based units 
is done on training ranges in Eive, Virtual and/or Constructive (EVC) environments. The 
Infantry Immersion Trainer (IIT) located on Marine Corps Base Camp Pendleton 
(MCBCP), CA, is an example of a live and virtual operational training environment 
(Kobus, 2012). Eor this thesis research, the IIT on MCBCP served as the facility for 
observational data collection. Currently, IIT staff conducts subjective performance 
assessments, usually in the form of After Action Reviews (AARs). These AARs 
traditionally occur at the conclusion of a scenario or on an ad hoc basis at the discretion 
of the training unit (Kobus, 2012). While these subject matter experts (SMEs) provide 
critical expertise and insight, feedback tends to be limited (Kobus, 2012). Moreover, 
there are neither standardized metrics for evaluating the small unit’s behavior while 
engaging in scenarios, nor standardized processes for providing feedback (Kobus, 2012). 
Including behaviorally-anchored metrics in the training and evaluation process at the IIT 
will improve feedback to trainees. Measuring the small-unit’s performance in this 
context will improve assessments of decision-making in certain IIT scenarios, thus 
contributing to the standardization of Marine operational training (Kobus, 2012). 

This thesis research is segmented into two phases, the knowledge elicitation phase 
and the observational data-collection phase. Eor the knowledge elicitation phase (phase 
one), semi-structured interviews were performed with SMEs to elicit information relevant 
to developing measures to assess performance. These SME interviews were used to 
develop a set of observer-based performance measures used in specific IIT related 
scenarios as a proof of concept. 

The output from the knowledge elicitation phase (phase one) was used to develop 
a set of behaviorally-anchored metrics for use in evaluating the performance of Marine 

Corps squads engaging in training at the IIT. These metrics were used by the researcher 
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in the observational data collection phase to collect performance data at the IIT. The goal 
for this phase was to test the metrics developed in the first phase to determine if they 
provide objective and effective measures. Developing valid and reliable metrics will 
facilitate assessment of individual and collective performance and overall training 
effectiveness. “These measures will enhance and complement current subjective 
measures” (Kobus, 2012, p. 28). 

Live and virtual-training environments such as the IIT provide a new approach to 
instruct, train and evaluate marines and sailors. Compared to traditional Military 
Operations on Urban Terrain (MOUT) facilities and ranges, these new training ranges 
offer more dynamic and immersive experiences approximating realistic combat 
environments (United States Joint Forces Command, 2011). The objective for this thesis 
research was to provide the evaluator a means to quantify decision-making performance. 
The benefit from this research is that it provides instructors behaviorally-anchored 
measures that can be used to provide specific relevant feedback to trainees. Through 
realistic training. Marines will be prepared for complex and ambiguous situations by 
leveraging proficiency in technical and tactical knowledge, while executing sound 
judgment in a decentralized manner (Conway, 2008). 
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II. BACKGROUND 


A. HISTORY OF ASSESSMENTS 

The sinking of the British passenger liner RMS Lusitania by a German submarine 
in 1915 (Lauriat, 1915), and the unrestricted German submarine campaign of 1917 
(Tucker & Roberts, 2005) thrust the United States into the “War to End all Wars” of 
WWI (Jamieson, 1988, p. 101). Americans such as Robert M. Yerkes, president of the 
American Psychological Association (APA), were eager to aid the war effort. He quickly 
convened the APA in an effort to pool their collective knowledge, methods, and expertise 
in order to increase the efficiency and effectiveness of screening and assigning enlistees 
and officers to military positions (Kevles, 1968). Along with noted psychologists, Robert 
L. Thorndike, Lewis Terman, and David Wechsler, the APA created an alpha and beta 
assessment battery that would evaluate and then classify the intellectual potential of 
military enlistees and officer candidates (Kevles, 1968; Cardona & Elspeth, 2007). By 
1976, these initial assessment tools were refined, and then formally presented as the 
Armed Services Vocational Aptitude Battery (ASVAB). Currently, the Department of 
Defense (DoD) uses “this test battery for both screening qualified candidates and 
assessing trainability for classified jobs” (Dubay, 2007, p. 14). 

An assessment “determine[s] the importance, size, or value of’ a process by an 
individual or organization (“Assess,” n. d.). In traditional educational curriculums, 
assessments are characterized as having either a formative or summative format. The 
intent of both formats is to gauge the learner’s comprehension of the curriculum and 
validate the instructor(s) or institution’s method of delivering the education or training 
(Wiliam, 2000). The summative format is a judgment at a milestone in the learner’s 
knowledge acquisition process (Taras, 2005). The formative assessment entails an 
iterative loop that adjusts and corrects a learner’s gap or deviation in knowledge from an 
established knowledge standard (Taras, 2005). The intent of either assessment format is 
accountability to its internal (teachers, students, and parents) and external stakeholders 
(school boards, government, etc.) (Linn, 2000; Shepard, 2000A). 
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Historically, assessments have been used to reform a particular social policy, such 
as Head Start, after-school programs, school uniforms, or instructional strategy (Linn, 
2000). The physical implementation of assessments is typically relatively inexpensive, 
transparent, and easily implementable which provides a smooth promotion and inclusion 
into educational environments (Linn, 2000). Figure 1 illustrates the evolving nature and 
application of assessments with regard to curriculum, function, and learning theory. 
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Figure 1. The historical and evolving viewpoints of assessment, curriculum, and 

learning theory (From Shepard, 2000B, p. 4). 


Twentieth century approaches to assessment viewed instruction as a process to 
eliminate inefficiencies from a particular learner’s behavior (Shepard, 2000B). Under 
this paradigm, learners were expected to master all the basic domain knowledge through 
repetition exercises and thereby incrementally develop expertise (Shepard, 2000A). This 
linear approach to instructional assessment migrated into summative testing 
methodologies such as the Ohio Graduation Test (OGT) and the Scholastic Assessment 
Test (SAT) (List of Standardized Tests in the United States, n. d.). 

The instructional assessment domain is currently undergoing a transformation in 

response to the policies in the 2001”No Child left behind Act” (Kim & Sunderman, 2005; 

Linn, Baker, & Betebenner, 2002; No Child Left Behind Act of 2001 (2002)). 

Additionally, the congressionally mandated National Assessment of Educational Progress 

(NAEP) instituted a national standardization protocol that enforces state education 
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performance requirements for pupils in primary and secondary schools (Swanson & 
Stevenson, 2002). These rules are based upon egalitarian principles that are agnostic to 
geographic, socioeconomic, and/or linguistic differences between diverse populations 
(Kim & Sunderman, 2005). Instituting a carrot-and-stick approach to reforming 
education and assessment means educators, pupils, and schools are being held 
accountable for the educational outcome of the pupil. Therefore, primary and secondary 
education will ideally provide pupils “a fair, equal, and significant opportunity to obtain a 
high-quality education and reach, at a minimum, proficiency on challenging state 
academic achievement standards and state academic assessments” (NCLB, 2001, Sec. 
1001 [3]). 

The domain of learning theory, and more specifically knowledge development, is 
evolving with advances in medicine, science, and technology (Dede & Barab, 2009; 
Osborne & Wittrock, 1983). Traditional associative and behaviorist approaches to 
learning and instruction are giving way to newer mental models based on cognitive and 
constructivist research and their applied learning theories (Anderson, Reder, Simon, 
Ericsson, & Glaser, 1998). These emergent approaches are less concerned with solely 
eliminating inefficiencies in the learner’s behavior. Cognitive and constructivist 
approaches focus on tailoring knowledge development to the individual’s social and 
cultural learning biases and aptitudes (Shepard, 2000B). This movement in learning 
science has promoted the integration of formative assessment methodologies in addition 
to, or in place of, traditional summative regimes (Harlen & James, 1997; Shepard, 
2000A; Shepard, 2000B). 

B. HISTORY OF THE AFTER ACTION REVIEW 

The most valuable and impactful assessment used in operational training is the After 

Action Review, or AAR (Smith, Yates, Valdyke, Roby, & Denney, 2010). The AAR, as a 

training tool, came into existence approximately 45 years ago and was predicated upon two 

important influences (Morrison & Meliza, 1999). The first influence was S. L. A. Marshall’s, 

“Island Victory: The Battle of Kwajalein Atoll.” In this book, he employs “the interview after 

combat” method, where he recalls the “very words of the men who fought - and therefore a 

highly accurate account of exactly what happened, as one scene of stress and confusion 
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followed another” (Marshall & Dawson, 1944, p. xxi; cited in Morrison & Mehza, 1999). The 
other influence was the “performance critique” method (Morrison & Meliza, 1999). This was 
the primary means of delivering performance feedback prior to the development of the AAR. 
This was originally employed to determine battle damage assessments during simulated battles 
where the outcomes of the exercises were subjectively adjudicated by “human umpires” 
(Morrison & Meliza, 1999, p. 6). In the early 1970s, the performance critique method was 
determined to be an ineffective means to assess training units and enhance their performance 
goals. From this coalescence, the DoD developed the AAR methodology to provide feedback 
in a more constructive and non-punitive manner (Morrison & Meliza, 1999). 

C. AAR AND MILITARY TRAINING APPLICATIONS 

In the military context, formative assessments are congruent with the AAR format 
(Morrison & Meliza, 1999; Rudolph, Simon, Raemer & Eppich, 2008). The AAR and its 
feedback mechanism afford individual and group learners a means to correct deviations 
in either an informal or instrumented (formal) manner (Kobus, 2012). In either AAR 
format, there must be a blending of training and assessment feedback that the instructor 
must weave into a narrative that the learner understands and then is responsive toward 
(United States Marine Corps Training and Education Command (TECOM), 2011). 
“Education’ teaches [the learner] how to think, perceive, and make decisions while 
‘training’ teaches what to think and do to make decisions” (TECOM, 2011, p. 18). The 
AAR must not be the means to solve the learner’s “problem” or train to the explicit 
solution. The formative assessment format must allow the learner to perceive, 
comprehend, and then innovate beyond their friction and allow a suitable range of 
alternative actions, instead of a single “correct” decision (Kobus, 2012, p. 26-27). 

In the AAR, the trainer must balance the desire to explicitly correct deviations or 
errors, thus circumventing the decision-making process of the learner. Instead, the trainer 
should ignore minor deviations that do not impact the final decision process and delay the 
expected errors that the learner has consistently made by offering timely, open-ended 
hints or leading questions. Only when delaying expected errors fail should the instructor 
intercede with a prompt that initiates the learner into self-assessment and correction. 

Depending on the constraints of the exercise, the trainer could engage the learner more 
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directly with explicit feedback on performance. This tactical change of explicitly 
interceding and guiding the training unit to a desirable end state would normally be 
driven by external real-world pressures (Lepper & Woolverton, 2002; Shepard, 2000B). 

1. ASSESSMENTS ON MILITARY TRAINING RANGES 

Currently, the DoD uses the AAR to evaluate trainee performance involving 
military training ranges (Frank et ah, 2004). Operational training in LVC environments 
is advantageous to training conducted in expeditionary environments because of the 
prohibitive expense, and unmitigated risk to personnel, materiel, and/or resources (Ford 
et ah, 2004; Boese, 2013). The USMC executes the majority of its operational training 
on air, naval, and/or ground training ranges throughout the continental United States 
(CONUS) and territories (Under Secretary of Defense (Personnel and Readiness), 2012). 
Additionally, selected training ranges in friendly foreign host nations are utilized to 
promote friendly military relations and enhance coalition training capabilities (Sanders, 
1998). Military use of assessments and the operational training conducted on various 
training ranges share a common functional and mutually dependent end state (Lanman, 
Becker, & Samper, 2009). 

a. USMC Training Ranges 

The current range infrastructure utilized by the USMC is characterized by 
dispersed geographic locations, dissimilar capabilities and capacities with the 
requirement to support a broad spectrum of mission essential warfighter training 
(TECOM, 2006). Training ranges are classified by their particular size and/or level of the 
training unit (smallest to largest); specifically, individual level, unit level. Marine Air 
Ground Task Force (MAGTF) and Marine Expeditionary Unit (MEU) Eevel, and 
MAGTE Marine Expeditionary Battalion (MEB) Level (TECOM, 2006, p. II). Their 
central purpose and function is to train Marines collectively in the approximated 
operational environment that enables effective operations in complex and uncertain 
expeditionary environments (Conway, 2008). Euture reference to Marine Corps training 
ranges will be specific to air/ground training ranges located in CONUS and Hawaii. 
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b. Infantry Immersion Trainer 

The original infantry immersion trainer (IIT) is located on training range 
“Area 62” on Marine Corps Base Camp Pendleton (MCBCP) (Babb, 2007). The IIT was 
conceptualized by then I Marine Expeditionary Force (I MEF) Commanding General 
James N. Mattis. He wanted and demanded industry, governmental agencies, and 
academia to construct a training environment that would prepare Marines for 
deployments to Iraq in support of Operation Iraqi Freedom (OIF). His training intent for 
the IIT was for Marines to be immersed with the “sights, sounds, smells and chaos of 
urban and close quarters battle so that their first real firefights were no worse than their 
simulations” (Muller, Schmorrow & Buscemi, 2008, p. 14). 

In 2007, with support from the Office of Naval Research (ONR), Marine 
Corps Warfighting Faboratory (MCWF), and I MEF staff, the IIT concept became an 
operational reality (Muller et ah, 2008; Schwetje, 2009). The central mission of the IIT is 
to “provide the most realistic combat conditions and settings to the small unit, in a virtual 
format, in order to teach and reinforce combat decision-making and small unit mission 
rehearsal” (Babb, 2007). Currently, there are two additional IIT facilities located on 
MCB Camp Fejeune, NC, and MCB Kaneohe Bay, HI. Training operations commenced 
at these installations on July and October, 2011, respectively. Both facilities operate 
independently of the other IITs with regard to their training concept of operations 
(Kobus, 2012). All future reference to the IIT will be specific to the MCBCP IIT. 

c. Assessments at the Infantry Immersion Trainer 

Currently, assessments of unit performance during operational training 
exercises, particularly at the IIT, tend to be subjective (Kobus, 2012; Matthews & Beal, 
2002). Moreover, there are neither standardized metrics (tools or applications) for 
evaluating the small unit’s performance while engaging in scenarios nor standardized 
processes for providing objective feedback (Hannigan et ah, 2012; Kobus, 2012). 
Additionally, IIT trainers conduct performance assessments on an ad hoc basis. While 
these subject matter experts (SMEs) provide critical expertise and insight, feedback tends 
to be limited in nature. The trainers debriefing narrative identifies “what happened 
during the scenario, why it happened, and how it should be done in similar situations” 
(Kobus, 2012, p. 24). 
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IIT trainers and staff utilize different assessment techniques for the pre¬ 
training, training, and post-training exercise phases. In the pre-training phase (prior to 
training execution), situational judgment tests (SJTs) are used to assess trainee courses of 
actions (COAs) via scenario-based problems to resolve training effectiveness issues 
(Kobus & Viklund, 2012). Training effectiveness issues pertain to “decision-making, 
tacit knowledge and practical ‘know-how’ domains” (Kobus, 2012, p. 28). During the 
training phase, trainers provide “as needed” performance assessments via declarative 
verbal statements. This feedback is provided by an individual lead trainer or through 
multiple trainers working in teams (2-3 trainers). Prompts providing guidance are 
normally provided while walking alongside the training unit (Kobus, 2012). During the 
post-training phase, formative assessments are conducted in one of four AAR formats: 
Hot Wash, Informal AAR, Contour Camera instrumented AAR, or tactical video¬ 
capturing system (TVCS)/Video Flashlight instrumented AAR (Kobus, 2012; “Contour 
(camera system),” n. d.). These formats are described in the next section. 

d. After Action Review at the Infantry Immersion Trainer 

IIT trainers use four AAR formats to assess and debrief the training units’ 
performance. The least formal, the Hot Wash, is generally performed when a more 
structured method is not required, usually with unit members forming a circle formed 
around the trainer. The trainer briefly provides his evaluation regarding the unit’s 
performance by highlighting his positive and negative observations of the exercise. This 
is followed by the trainer explaining corrective actions for observed faults regarding the 
unit’s tactics, techniques, and procedures (TTPs). This type of AAR usually takes no 
longer than 10 minutes (Kobus, 2012). 

The informal AAR extends the Hot Wash with a few notable additions. 
This method usually employs a 3-D block model to visually depict the eagle’s eye view 
of the training area. The alternative to the 3-D block model is a projection of the training 
area on a television screen or computer monitor (Kobus, 2012). The training unit 
members, particularly the individual squad and/or platoon members, are the focus of this 
type of AAR. The trainer poses probing questions to elicit feedback from all members of 
the unit in order to share and hear all relevant perspectives pertaining to critical errors 
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observed in the exercise. The goal of having each member share their perspective is to 
openly discuss their thought processes and decision-making actions (Department of the 
Army (DoA), 1993; Kobus, 2012). This type of AAR usually requires no longer than 20 
minutes (Kobus, 2012). 

The next two types of AARs, the Contour Camera instrumented AAR and 
TVCS/Video Flashlight instrumented AAR only differ regarding the way they collect 
video and audio, the system’s processing time, and viewing perspective (“Contour 
(camera system),” n. d.; Kobus, 2012). The TVCS/Video Flashlight and Contour Camera 
instrumented AARs are both conducted in the same structured classroom environment 
(Kobus, 2012). The Video Flashlight systems and TVCS are specifically used during 
phase I and phase II, respectively (Kobus, 2012). The Contour Camera approach can be 
used in either phase I or phase II. In the TVCS/Video Flashlight method, independent 
camera operators and video technicians use a closed-circuit television (CCTV) computer 
network to capture video streams from myriad cameras throughout the phase I/II training 
areas. Video technicians and camera operators work closely with trainers who are out 
with the training units, to mark and save important video selections to be used in 
PowerPoint presentations in the AAR. The total video processing time required, before 
the data can be used for the AAR, ranges from 20-50 minutes after each run, a significant 
delay which impacts the number of training runs that can occur in a day (Kobus, 2012). 

The Contour Camera method is a different approach to capturing video. 
The trainer employs a small handheld camera and selects only those relevant video 
streams that will provide the maximum value to the training audience. The video streams 
are then directly downloaded from the memory of the Contour Camera to the classroom 
computer used for the AAR. Compared to the TVCS and Video Flashlight system 
processing time, the required time to completely process the selected video streams is 
approximately five minutes. Due to the operational requirements of the training units and 
the advantage presented by the Contour Camera’s processing time, the Contour Camera 
instrumented AAR method is now the preferred method of instrumented AARs. The 
training unit uses the TVCS/Video Flashlight instrumented PowerPoint presentations for 
archive and complementary training purposes. 


12 



D. CURRENT CAPABILITY GAP IN ASSESSMENTS IN THE DOD 

The USMC and the Department of the Navy have recognized specific gaps in the 
manner in which we evaluate and train our Marines and Sailors, particularly in live, 
virtual and constructive (LVC) environments. Both the Marine Corps and the Navy face 
difficulties in conducting objective evaluations of the effectiveness of training systems 
and training environments in terms of achieving training objectives. This ability to 
objectively evaluate Marines and determine their level of mastery of the subject matter 
has been identified as a gap in current performance assessment methods that include an 
AAR or in the assessment process in general (Kobus, 2012; ONR, 2011). 

Including behaviorally-anchored rating scales to objectively measure if, or how 
well, training objectives were met is one method to overcome this gap. These scales can 
be used to measure the performance of the training unit and the results can either be 
provided during the AAR or reviewed by the unit leader independent of the AAR 
process. This discretion afforded to the training unit and its operational tempo provides 
flexibility within the unit leader’s training timeline. However, the more effective method 
of quickly assessing the unit’s level of mastery of training objectives would involve 
incorporating this feedback as part of the AAR. Development of valid and reliable 
measures and metrics will facilitate the assessment of individual and collective 
performance and overall training effectiveness. “These measures would enhance and 
validate current subjective measures” (Kobus, 2012, p. 28). 

E. MOTIVATION AND PURPOSE OF RESEARCH 

Research in the domain of performance assessment and its impact on Marines was 
motivated by a strong desire to have a positive effect on the Marine Corps. Aligning this 
interest in performance assessment with a documented Marine Corps gap, adheres to the 
CMC’s guidance that Naval Postgraduate School (NPS) Marines conduct their thesis 
research in areas that “align with the USMC needs” (CMC, 2010A, p. 14). The objective 
of this research was to provide military trainers with behaviorally-anchored measures that 
can be used to provide specific feedback to trainees. Research conducted for this thesis 
has contributed to developing tools and methods to quantifiably assess the small-unit 
leader’s decision-making abilities. 
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F. BENEFITS OF RESEARCH 

Live and/or virtual-training environments such as the IIT provide a novel 
approach to train and evaluate Marines and Sailors in a dynamic, immersive environment 
approximating realistic combat environments (Smith et al. 2010). The results of this 
research provide the trainer with measures that can be used to quantify decision-making 
performance. Current budgetary conditions experienced by military and civilian leaders, 
necessitate an effort to maximize efficiencies in existing programs and organizations 
(Bilmes, 2013). Inculcating the lessons learned from operational training exercises into 
subsequent training evolutions will afford greater skill retention in the individual and 
collective warfighting skill sets and use military budgets in a more efficient manner 
(Boese, 2013). 
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III. LITERATURE REVIEW 


A. COGNITIVE AND CONSTRUCTIVIST PSYCHOLOGY 

In the online article “More Complex Than a Galaxy” - New Insights Into the 
Enormous Biochemical Complexity of the Human Brain, Sir Roger Penrose was 
referenced comparing the complexity of the human mind to that of the galaxy. He said, 
“consider the human brain[,] if you look at the entire physical cosmos, our brains are a 
tiny, tiny part of it. But they’re the most perfectly organized part. Compared to the 
complexity of a brain, a galaxy is just an inert lump” (as cited in www.dailygalaxy.com, 
2012). To better understand the human mind, modern technology and science have 
deconstructed the brain’s organic matter into its biochemical and genetic underpinnings 
(Hedden & Gabrieli, 2004). While these endeavors have advanced our contemporary 
understanding of the human mind and its capabilities, there is a plethora of outstanding 
questions regarding its cognitive processes (McGinn, 1989). Psychology was developed 
as a means to comprehend these cognitive processes, the mind’s relationship to human 
behavior, and its connection with the external environment and society (Anderson, 2009). 

Our modem understanding of the human mind and cognition came from centuries 
of philosophical literature, theoretical and empirical research, and advancements in 
technology and neuroscience (Finger, 2001). Ulric Neisser formali z ed the domain of 
cognitive psychology with his 1967 book, “Cognitive Psychology” (Neisser, 1967). 
Neisser’s work was fundamentally influenced by Noam Chomsky’s 1959 critique of B.F. 
Skinner’s “Verbal Behavior” (world’s leading behaviorist theorist’s book on language). 
In his assessment of Skinner’s work, Chomsky provided an “effective refutation of the 
behavioristic approach to language” (Neisser, 1967, p. 246). Neisser presented 
cognitivism as a more suitable and holistic approach to knowledge elicitation. He 
emphasized that a strict adherence to behavioral response or association to external 
stimulus was insufficient to explain the mind’s mental processes (Neisser, 1976). 

Nessier defined cognition as the “sensation, perception, imagery, retention, recall, 
problem-solving, and thinking” aspects of mental processes (Neisser, 1967, p. 4). The 
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cognitivist “emphasize[s] individual thinkers and their isolated minds” in the absence of, 
“the social nature of cognition and meaning” (Jonassen & Land, 2000, p. 26). A central 
tenet of cognition is metacognition or “Knowing about Knowing” (“Metacognition,” n. d.). 

Metacognition refers to one’s knowledge concerning one’s own cognitive 
processes or anything related to them, e.g., the learning-relevant properties 
of information or data. For example, I am engaging in metacognition if I 
notice that I am having more trouble learning A than B; if it strikes me 
that I should double check C before accepting it as fact. (Nisbet & 
Shucksmith, 1984, p. 6) 

Metacognition is evident when learners engage in training involving ambiguous 
scenarios with insufficient information. Effective outcomes are achieved when learners 
comprehend what is necessary to overcome the obstacle(s) to achieving their objectives 
and adapt their decision-making strategies accordingly (Morrison & Fletcher, 2002; 
Fortunate, 1991). Training conducted at the infantry immersion trainer (IIT) Camp 
Pendleton, CA, employs this cognitivist approach (Kobus & Viklund, 2012). 

Cognitivist theory evolves into the Constructivist paradigm by incorporating the 
internal symbolic meaning of external social and environmental relationships. The 
constructivist rejects the notion that the learner perceives reality from the external 
environment; rather reality is internally constructed based upon the learner’s perceptual 
experiences (Cooper, 1993). “Understanding is an individual construction, we cannot 
share understandings but rather we can test the degree to which our individual 
understandings are compatible” (Savery & Duffy, 1995, p. 1). During after action 
reviews (AARs), Marines verbally recollect the events of the scenario and present their 
interpretations of reality. Providing multiple facets of understanding and perspective 
enhances the shared understanding of events. This dissemination of various perspectives 
fosters development of common situational awareness among the individual Marines and 
affords them the opportunity to iteratively correct their misinterpretations of events 
presented in the training scenario (Smith et ah, 2010). 
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B. ADDITIONAL IMPLICATIONS OF COGNITIVE AND 

CONSTRUCTIVIST PSYCHOLOGY 

Cognitive and constructivist science naturally apply to the fields of educational, 
abnormal, social, developmental, and personality psychology (“Cognitive Psychology,” 
n. d.). These specializations within psychology provide a broader applied discipline 
involving “the scientific study of the behavior of individuals and their mental processes” 
(“Psychology,” n. d.). Due to the limited scope of this thesis, the focus of this chapter 
will be on cognitive and constructivist influences on educational and social psychology. 

1. Educational Psychology 

In instructional environments, learners and instructors inherently possess a variety 
of biases, aptitudes, experiences, and sociocultural backgrounds. Instructional design of 
the learning environment and the relevant cognitive learning theory must be tailored 
according to these individual differences (Bereiter, 1990; Brook & Ross, 2002). This 
dynamic, current learning theory is divided into three levels of learning strategies. At the 
lowest level are learning theories based on genetics and neuroscience. Theories based on 
a psychological context constitute the medium level and philosophical and secular based 
contexts comprise the most complex and highest level of learning theories (Brook & 
Ross, 2002). Categorization in this manner aids facilitation of an optimal learning 
strategy, however the strategy does not include the additional and necessary external 
relationships included in constructivism. The intention of this discussion is not to 
highlight the differences or refute the merits of cognitivism; rather the goal is to 
emphasize their complementary natures (Bereiter, 1990). Both the cognitivist and 
constructivist approaches present an end state where the learner perceives, analyzes, 
comprehends and then decides on suitable courses of actions (COAs) (Gelder, 2005). The 
optimal outcome of either approach is an effective byproduct that is efficiently employed 
by the instructor. 

2. Social Psychology 

Humans are intrinsically social creatures and are beholden to societal norms and 
traditions. Comprehending verbal and non-verbal interactions (such as, gesture, body 
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language, posture, tone of voice or facial expressions) is essential to form social identity 
and relationships with individuals and groups (Howard, 2000). Social psychology is “the 
branch of psychology that studies the effect of social variables on individual behavior, 
attitudes, perceptions, and motives [, and] also studies group and intergroup phenomena” 
(“Social Psychology,” n. d.). Social psychologists attempt to understand individual and 
group social interactions through cognitivist and constructivist paradigms (Reich, 2003; 
Reicher, 2004). 

The concept of identity is critical to social standing and purpose within the group. 
Organized individuals have a cognitive need to efficiently categorize individuals, groups, 
events, and objects in order to reduce their cognitive workload (Howard, 2000). In 
highly organized groups such as Marine squads and platoons, comprehending each 
Marine’s role and function is essential to mission accomplishment. These classifications 
aggregate complex and fluid relationships into discrete elements which aid in recollection 
and utility. Marine units operating in highly complex and/or ambiguous situations rely 
on this identity simplification as a means to mutually support common objectives 
(Zaccaro, Rittman, & Marks, 2002). 

Cognitive readiness [refers to] the mental preparation (including skills, 
knowledge, abilities, motivations, and personal dispositions) an individual 
needs to establish and sustain competent performance in the complex and 
unpredictable environment of modern military operations. (Morrison & 
Fletcher, 2002, p. ES-1) 

Within the cognitive readiness domain, there are generally accepted 
characteristics or heuristics that support effective decision-making. The following 
paragraphs describe the United States Marine Corps (USMC) cognitive readiness 
initiatives related to decision-making at the small unit level. 

3. United States Marine Corps Cognitive Readiness Initiatives and 
Research 

In 2008, Commandant of the Marine Corps (CMC), Gen. James T. Conway 
(USMC), promulgated his “vision of the future Corps and a plan for creating the Marine 
Corps of 2025” (Conway, 2008, p. 3). This document informed Marines, combatant 
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commanders, and civilian leadership of the future direction the Marine Corps would 
provide to national defense. Additionally, it prioritized strategic objectives in order to 
“respond rapidly and decisively to crises anywhere in the world” (Conway, 2008, p. 14). 
One of these objectives established the need to train, educate, and prepare Marines for 
expeditionary operations in challenging environments. This Marine Corps document 
maintains that additional resources provided to noncommissioned (NCO) and junior 
officer grades will assist acclimatization to decentralized combat operations. This 
professional military education will condition ethical and sound decision-making skills in 
small-unit leaders. 

The Marine Corps Vision & Strategy 2025 compendium is a planning document 
that will be promulgated in the following years in order to bring the CMC’s intent from 
concept to reality. The implementation plan organized this strategy into three future 
years defense plan groups: “near-term: present [2008] - 2011; mid-term: 2012 - 2016; 
and far-term: 2017 and beyond” (Conway, 2008, p. 1). The document’s first task directed 
the development of diverse approaches to NCO and junior officer intuitive decision¬ 
making. Consistent with Klein’s naturalistic decision-making paradigm, small unit 
leaders will be trained to effectively recognize cues and patterns from the cultural and 
environmental terrain in order to acquire expertise. Current and emergent decision¬ 
making initiatives were codified in this implementation guidance (Conway, 2008). 

In January 2011, the small unit decision-making (SUDM) initiative, under the 
purview of United States Marine Corps Training and Education Command (TECOM), 
conducted a workshop that focused on identifying tools, techniques, and procedures to 
support the SUDM goals (TECOM, 2011). The goal of the SUDM initiative is to 
accelerate cognitive readiness in novice small-unit leaders and inculcate its tenets in their 
subordinates (Schatz, et ah, 2012). The SUDM workshop brought together leading 
cognitive researchers, scientists, and academics along with Marine officers who work in 
PME development and instruction. Workshop participants collectively agreed that the 
SUDM initiative and its goals would only be achieved through sustained planning and 
commitment. Additionally, they identified five competencies of cognitive readiness that 
support effective decision-making. 
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As listed and defined in Table 1, these cognitive readiness competencies 
(indicated by the shaded cells) are sensemaking, adaptability, attention control, 
metacognition, and problem solving (Schatz, et al., 2012; TECOM, 2011). Table 1 also 
includes definitions for ten cognitive and relational skills (CARS) (indicated by the non- 
shaded cells) which represent a finer resolution of the five cognitive competencies and 
“indicate decision-making proficiency” (Ross et al., 2012, p. 1-2; TECOM, 2011). Each 
of the cognitive competencies and CARS represent different dimensions of cognitive 
readiness. Each attribute addresses a particular gap of decision-making during military 
operations. 
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Table 1. TECOM Small Unit Decision-Making Cognitive Competencies and CARS 

(From Ross et al., 2012, p. A2-A14). 

Sensemaking: The cognitive process, driven by a specific goal, of filtering information 
for relevancy and using it to construct and continually assess an explanation of the broad 
or specific situation, often in the form of a story, in order to understand how and why the 
situation evolved and anticipate what might happen next._ 

Perspective Taking: Visualizing the situation from another’s viewpoint and 
assessing his or her motivations and objectives, to predict his or her future actions and 
proactively position for or take advantageous action. _ 

Analytical Reasoning: Critically and deliberately examining, assessing, and 
critiquing one or more alternatives or assumptions in the context of specified goals 
(e.g., the mission) and against a set of evaluative criteria (e.g., intent, timing, 
resources, or ROE). _ 

Anomaly Detection: Critically and deliberately examining, assessing, and critiquing 
one or more alternatives or assumptions in the context of specified goals (e.g., the 
mission) and against a set of evaluative criteria (e.g., intent, timing, resources, or 
ROE). _ 

Situational Assessment: Analytically or intuitively identifying and collecting 
information from multiple available sources, including one’s own knowledge, to 
analyze relevant factors of METT-TC (mission, enemy, terrain and weather, troops 
and support available, time available, civil considerations) and construct an 
understanding of the situation to support a specific task or goal. _ 

Adaptability: Fluidly modifying or changing one’s planned actions when the situation 
has changed from what was expected, or when the typical approach or plan is rendered 
less effective than necessary._ 

Situational Assessment: Analytically or intuitively identifying and collecting 
information from multiple available sources, including one’s own knowledge, to 
analyze relevant factors of METT-TC and construct an understanding of the situation 
to support a specific task or goal. _ 

Cognitive Flexibility: Applying knowledge and principles of tactics and leadership 
differentially based upon the unique demands of the situation. Applying knowledge 
learned in one context to multiple relevant contexts. _ 

Ambiguity Tolerance: The ability to calmly withstand and operate within uncertain 
environments by delaying drawing a conclusion or making a decision, or by making 

assessments and decisions in the face of uncertainty. _ 

Continued on next page 
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_ Continued from previous page _ 

Attentional Control: Activities related to maintaining a focus on mission completion 
despite distracters including stress, boredom, fatigue, and emotion._ 

Resilience: Overcoming the stress, fatigue, emotion, or pain associated with a current 
or past event or situation in order to maintain or return to effectiveness as a leader and 
decision maker. _ 

Self-Regulation: Monitoring, assessing, and adjusting one’s own behavior and its 
effects in order to impact the situation in a way that supports mission, unit, or training 
goals. _ 

Metacognition: Activities related to considering one’s own thought processes, including 
assessments of strengths and limitations or developmental needs, in support of 
performing or learning the job._ 

Self-Awareness: Conscious knowledge of one’s own character, motives, knowledge 
base, and skill set in order to request information or assistance when the requirements 
of the situation call for capabilities beyond one’s current abilities. _ 

Problem Solving: Identification, definition, examination, prioritization, and resolution of 
situations that impede task or mission accomplishment._ 

Analytical Reasoning: Critically and deliberating examining, assessing, and 
critiquing one or more alternatives or assumptions in the context of specified goals 
(e.g., the mission) and against a set of evaluative criteria (e.g., intent, timing, 
resources, or ROE). _ 


22 









Proficiency in each of the cognitive competencies and CARS will enhance and 
support the decision-making performance of the individual unit leader (Ross et ah, 2012). 
Performance in this context does not exclusively apply to the training and/or garrison 
environment. Increasing the unit leader’s performance in the training environment must 
have a positive correlation with expeditionary operations (Morrison & Fletcher, 2002). 
Attaining a level of competence in cognitive readiness should have an equal impact on 
operational effectiveness (Walsh & Shingledecker, 2006). Cognitive readiness is viewed 
as an interventional process which is assessed and measured against its product (Morrison 
& Fletcher, 2002). 

C. DECISION-MAKING THEORIES 

Cognitive readiness is a concept that provides a set of attributes that can be used 
to prepare Marines to function effectively given the external influences from the 
environment. The decision to act or not act in operational environments is pivotal given 
the life and death nature of combat. Decision-making theories promote an understanding 
of individual and organizational decision-making processes with respect to their 
outcomes. Decision-making theories discussed in the remainder of this chapter will focus 
on intuitive strategies. 

In decision-making situations that are extremely complex, ambiguous, or time- 
constrained, a more streamlined decision-making process is typically employed. The 
rational decision-making approach is not applicable for highly uncertain or time 
constrained decision-making situations. In fast-paced, uncertain decision-making 
situations a “satisficing” decision strategy is employed (Simon, 1957, p. 204-205). 
Satisficing reduces the learner’s cognitive workload by shedding non-essential tasks from 
consideration and allows the learner to focus on the first sufficient COA rather than the 
optimal. The learner’s pattern recognition capability is an important discriminator 
between the decision-making abilities of a novice and that of an expert. The following 
discussion of decision-making theories is not meant to be all-inclusive; rather theories 
discussed in the following sections are considered to be most applicable to this thesis 
research. 
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1. Prospect Theory 

Through much of the 20th century, decision-making theory was dominated by 
expected utility theory (EUT) (Kahneman & Tversky, 1979). EUT, postulated by 
Bernoulli in 1738, and later refined in 1947 by Von Neumann-Morgenstern, explained 
individuals in terms of the “rational decision maker” (“Expected Utility Hypothesis,” n. 
d.; “Von Neumann-Morgenstern utility theorem,” n. d.). The rational or normative 
model of decision-making described how individuals probabilistically decided on the 
most preferred action given all the information. While concise and explanatory, the 
theory’s rationale did not provide an adequate explanation for decision-making situations 
that involve uncertainty and preference. In 1979, Kahnman and Tversky published a 
critique of EUT through their prospect theory of decision-making. This theory’s critique 
of EUT was based on empirical examples of economic behavior related to gambling and 
insurance (Kahneman & Tversky, 1979). Prospect theory fundamentally contributed to 
decision-making and “demonstrate[d] that human judgment arises from qualitatively 
different processes than suggested by normative theories, and argue[d] that heuristics 
would provide a better starting point for the development of psychological theory” 
(Phillips, Klein, & Sieck, 2004, p. 297-298). 

2. Recognition-primed Decision Model 

In 1989, the recognition-primed decision (RPD) model was introduced by Klein, 
Calderwood, and Clinton-Cirocco. The model explains how people make decisions 
expeditiously without comparing multiple options. It describes a naturalistic method of 
recognizing the “plausible [and suitable choice] as the first one to consider” (Ross, Klein, 
Thunholm, Schmitt, & Baxter, 2004, p. 6). Klein and associates developed the RPD 
model after observing experienced firefighters and interviewing them regarding their 
decision-making processes. They discovered that these firefighters never made explicit 
decisions; instead they instinctually acted upon the ground truths of the situation (Phillips 
et ah, 2004). That is to say, firefighters were able to recognize key aspects of the 
situation, based on prior experience with similar situations, and were able to come up 
with a workable solution. Through experiential recognition, firefighters recalled previous 
similar events, their associated solutions, and implicitly decided upon the most workable 
COA. 
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As depicted in Figure 2, two critical components of the RPD model are situation 
assessment and mental simulation. Both processes provide the means to develop a 
particular COA and assess its validity. The pedagogical intent of RPD is to condition 
individuals to effectively synthesize meaningful connections from previous experiences, 
and provide timely and efficient correlations to the current solution space. When the 
decision maker uses a RPD strategy he/she does not systematically process all 
combinatorial paths or probabilities in order to rationally decide; rather using a RPD 
decision strategy entails selecting a COA that suffices given the limitations and 
constraints of the real-world (Klein & Klinger, 1991). 


A. Simple Match 


^xperience the Situatior^ 


B. Developing a Course 
of Action 


C. Complex RPD Strategy 




Figure 2. RPD Model (From Klein and Klinger, 1991, p. 18). 


3. Observe-Orient-Decide-Act Loop 

In 1977, the observe-orient-decide-act (OODA) loop, or Boyd’s theory, was first 
presented in Col. John R. Boyd’s (USAF), “Patterns of Conflict” (Osinga, 2005, p. 2; 
Boyd, 1986). This theory represented the culmination of Boyd’s work regarding rapid 
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decision-making in uncertain and competitive environments. Boyd’s theory describes the 
individual’s decision-making cycle as the means to adapt, maneuver, and survive in a 
belligerent and resource-constrained environment (Osinga, 2005). Boyd describes this 
competition as a contest of wills, where participants prove to be successful or 
unsuccessful relative to the timeliness and acuity of their decisions (Boyd, 1976). 

As depicted in Figure 3, during the observation stage, perceived sensory 
information from the external environment is processed and provides the necessary input 
for the orientation stage. The individual’s experiences, biases, and background (physical, 
moral, and cultural) provide the necessary contextual reference and meaning for that 
information. The synthesized product is then forwarded to the decide stage, where 
judgment and deliberation are used to either reject or accept the COA. Once the COA is 
implemented, subsequent feedback loops ensure the actions are aligned with the decision 
made (Ullman, 2007). 


Observe Orient Decide Act 



Figure 3. Boyd’s OODA Loop (From Ullman, 2007, p. 2). 


4. Cognitive Observe-Orient-Decide-Act Model 

The cognitive observe-orient-decide-act (C-OODA) model is a natural extension 
of the OODA Loop model. As shown in Figure 3, Boyd’s model was a “high-level 
abstraction of the decision cycle” (Hannigan et ah, 2012, p. 6). The critical limitation of 
the OODA Loop was its presentation of the cognitive decision-making process. The 
original model was deficient in ascertaining the necessary cognitive requirements used to 
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identify the “dynamic properties of the decision-making task” (Breton & Rousseau, 2007, 
p. 243). Additional limitations regarding solitary input channels and granularity of 
decision-making expertise required further research (Hannigan et ah, 2012). 

The C-OODA model resolves the limitations noted above by integrating three 
applicable cognitive theories into the observation, orientation, and decision phases of the 
OODA model: (the act phase is purposely excluded due to the additional resource and 
s ki ll requirements) (Hannigan et ah, 2012). The C-OODA loop integrates Treisman’s 
feature integration model, Endsley’s model of situation awareness, and the RPD model as 
the observation, orientation, and decision phases respectively (Endsley, 1995; Hannigan 
et ah, 2012; Klein, 1988; Treisman & Gelade, 1980). As depicted in Eigure 4, the C- 
OODA loop incorporates a higher- and lower-level of cognitive processing. The lower- 
level processing applies uniformly to all skill levels (novice to expert decision makers), 
while the higher-level processing is apparent when proficiency and expertise mature 
(Hannigan et ah, 2012). 



Eigure 4. C- OODA Eoop (Erom Hannigan et ah, 2012, p. 7). 

As depicted in Eigure 4, lower-level mental processing of perceived sensory 
stimuli occurs through feature matching and experiential association. Correlated objects 
(information) of interest are filtered out and forwarded to the next phase of the C-OODA 
loop where further processing of the objects will occur (Treisman & Gelade, 1980). 
Once the selected information is forwarded to the orientation phase, lower-level 
processing and comprehension of the information’s importance, priority, and relevance 
occur (Endsley, 1995). Once the information is comprehended, and then referenced with 
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respect to the individual’s norms, goals, and biases, relationships are established to 
facilitate projection of the individual’s knowledge. Finally, information is forwarded to 
the decide phase, where the decision maker becomes cognizant of the information’s 
relationship with respect to its importance or familiarity (Klein & Klinger, 1991). Once 
that understanding is achieved, individuals must evaluate the potential COA and the end 
state it will produce. Once this COA’s evaluation process is complete and the COA’s 
decision criterion is met, the COA is forwarded to the act phase. 

5. Maneuver Squad Leader Mastery Model 

The Maneuver Squad Leader Mastery Model (MSLMM) is a “five-stage 
descriptive model” that is “designed to (1) provide insights into how individuals 
progressively develop into high performing maneuver squad leaders, and (2) provide 
implications for what should be assessed and how during development” (Ross et ah, 
2012, p. 2). The model was developed under TECOM sponsorship for the SUDM 
initiative. The goal was to explain the process of optimizing attributes of expertise and 
decision-making competence as a “maneuver squad leader” (Ross et ah, 2012, p. 1). The 
model can be used to assess the maneuver squad leader (MSL) across nine performance 
areas via five cognitive competencies and ten CARS (see Table 1.). 

Behaviorally-anchored rating scales (BARS) that classify observable behavior 
with quantitative scales are incorporated in the MSLMM (Ross et ah, 2012). The 
MSLMM addresses the need to describe the decision-making process with respect to the 
maneuver squad leader through incremental stages (novice to expert). BARS used in the 
MSLMM were generated through the situational judgment test and decision requirements 
interview instruments. The MSLMM requires future pilot studies to refine the 
assessment battery (incorporating BARS) beyond the currently restricted academic 
domain (Ross et ah, 2012). 

D. PERFORMANCE MEASURES DEVELOPMENT 

The goal of this thesis is to develop performance measures that provide the 

evaluator a more objective means than currently used to assess the decision-making of 

Marine squad leaders in training environments. Assessing individual and unit 
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performance is “an essential part of team training and training research” (Shanahan, Best, 
Finch, & Sutton, 2007, p. 1). Factors that influence individual and unit performance and 
operational effectiveness can be qualitative or quantitative, singular or complex in nature 
(Schwab, Heneman, & DeCotiis, 1975). Examples of these factors that influence 
individual and unit performance are explained in the cognitive readiness section of this 
chapter. While the list of attributes in Table 1 is not all-inclusive, it does provide 
examples of cognitive and behavioral attributes used in the assessment and analysis of 
decision-making. 

This thesis views performance measures through the prism of individuals 
operating in organized units. Individuals effectively operating in organized units should 
exhibit competence related to “(1) information exchange, (2) communication, (3) 
supporting behavior, and (4) initiative and leadership” (Shanahan et ah, 2007, p. 5). 
These behaviors and the underlying cognitive processes that support these behaviors must 
be assessed within the context of the task, objective, and the environment where the task 
was performed (Phillips, Shafer, Ross, Cox, & Shadrick, 2006A). Diagnostically, the 
behavior must be distinct enough to be discriminable, while selective enough to allow for 
an acceptable amount of variability from the target audience (novice to expert). The 
behavioral assessment and its correlated metric must provide the evaluator the means to 
succinctly and efficiently annotate, audit, and catalog adherence to or deviations from the 
performance standard. Balancing the sensitivity and utility of the measurement tool 
requires due diligence in the design process, comprehension of the task and objective, 
and concurrence with subject-matter expertise (Phillips, Ross, & Shadrick, 2006B). 
Human interaction factors must be considered in the design and development of the 
measure along with implementation of the tool. The following sections on cognitive task 
analysis (CTA) and BARS describe the methodology used to develop performance 
measures for this thesis research. The subsequent method chapter will provide greater 
detail regarding these two phases. 
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1. Cognitive Task Analysis 

Cognitive task analysis (CTA) refers to a group of methods that are extensively 
used in cognitive systems engineering and naturalistic decision-making applications 
(Hutchins, Pirolli, & Card, 2004). Klein’s (2001) definition of CTA is “methods for 
capturing expertise and making it accessible for training and system design” (p. 173). 
Klein delineated the following five steps: (a) identifying sources of expertise, (b) 
assaying the knowledge, (c) extracting the knowledge, (d) codifying the knowledge, and 
(e) applying the knowledge. System design goals supported by CTA include human- 
computer interaction design, developing training, tests, and models to serve as a 
foundation for developing an expert system, and analysis of a team’s activities to support 
allocation of responsibilities to individual humans and cooperating computer systems. 
(An extensive treatment of CTA methodology appears in Crandall, Klein, & Hoffman, 
2006.) 

Different CTA methods are used for different goals (Hoffman, Shadbolt, Burton, 
& Klein, 1995.) My goal for conducting a CTA was to obtain operational expertise from 
SMEs on task performance of complex tasks that are trained in the IIT and use this 
information to develop behaviorally-anchored performance measures. A CTA requiring 
resource-intense analysis is characterized by complex, ill-structured tasks that are 
difficult to learn, complex, dynamic, uncertain, and real-time environments as well as 
multitasking. A CTA is appropriate when the task requires the use of a large and complex 
conceptual knowledge base, the use of complex goal-action structures dependent on a 
variety of triggering conditions, or complex perceptual learning or pattern recognition 
skills. The tasks performed by Marine Corps squads involve all of these characteristics. 

The CTA that was planned for this research would employ one of the two 
methods traditionally used for knowledge elicitation from SMEs. These techniques are 
the critical decision method (CDM) (Hoffman, Crandall, Shadbolt, 1998) and the 
knowledge audit (KA) (Militello & Hutton, 1998). The CDM generally asks experts to 
recall a critical incident and asks them to annotate decision nodes through a constrained 
timeline and draw out possible alternative vignettes through persistent inquiry. KA is a 

“set of questions or probes that have been extracted from the literature of expert-novice 
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differences” (Craig et al., 2012, p. 1027). The KA technique reveals noteworthy items in 
the subject matter, such as, a macro-level perspective of the events, meaningful signals 
from the environment, and how an expert would improvise when responding to presented 
vignettes. The KA provides expeditious data collection and implementation advantages 
with respect to CDM, but lacks the depth and detail that is required for CTAs not 
constrained in acute time and resource environments (Fowlkes, Salas, Baker, Cannon- 
Bowers, & Stout, 2000). The initial plan to conduct a CTA was altered. The ONR 
sponsor of this Marine Corps work directed our attention to a recent CTA report on lED- 
defeat (Ross et al, 2012). The research plan was modified with the intent to make use of 
the information included in the CTA on lED defeat 

2. Behaviorally Anchored Rating Scales 

Behavioral measures are defined as “performance dimensions and scale values in 
behavioral terms” (Schwab, Heneman, & DeCotiis, 1975, p. 550). These measures, when 
compared to qualitative trait measures, provide the evaluator more objective measures to 
assess the trainees’ performance. With measures that are less subjective, the evaluator 
can maintain the necessary objectivity during performance assessment. Behaviorally- 
anchored rating scales (BARS) were developed as a means to assess the individual’s 
performance in “multidimensional, behavior-specific terms” (Schwab et al., 1975, p. 
550). While the BARS concept is applicable to multiple domains, BARS developed for 
this thesis research are meant for “scenario-based, training sessions requiring 
sensemaking and/or decision-making, including [...] tactical decision game or decision¬ 
making exercise sessions” (Phillips et al., 2006A, p. 1). 

In developing BARS, there is a uniform methodology be used for their 
construction. Subject matter experts provide information regarding gradations of 
expertise (novice to expert) and efficacy (ineffective to highly effective) of outcome 
(Schwab et al., 1975). This knowledge elicitation procedure is normally conducted using 
the CTA method. Results of the CTA are used to develop, identify, classify, and group 
behavioral attributes according to the event. Once complete, a separate group of domain 
experts are asked to perform the identical identification, classification, and grouping 
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procedure with the goal of correlating specific behavior(s) to the events. The results are 
then compared and analyzed for behavior/event retention purposes. Concurrently, the 
latter group assigns efficacy ratings (point-scales) with respect to the behavior measures 
of performance. Typically, the developers agree on an acceptable level of variance in the 
behavior’s MoPs criterion, a statistical analysis is performed on the feedback, and those 
behaviors/events that are within the variance are retained. Finally, retained behaviors 
linked to the specific events are then incorporated into a “series of [horizontal] scales 
(one for each dimension) anchored by the retained incident [event]” (Schwab et ah, 1975, 
p. 551). Further refinement from continued BARS utilization will address practical and 
time-related constraints and limitations. 
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IV. METHODOLOGY 


This chapter describes two separate, but interrelated phases: the behaviorally- 
anchored rating scales (BARS) development phase and the infantry immersion trainer 
(IIT) final data collection phase. BARS were developed for the improvised explosive 
device (lED) defeat decision-making domain. The BARS development phase focused on 
the design and development of BARS for use at the IIT on Marine Corps Base Camp 
Pendleton (MCBCP), CA. Discussion of the BARS development phase also describes 
observational data collected during IIT research visits to test the initial set of BARS. 
Additionally, attributes indicative of decision-making proficiency levels were elicited 
during interviews with subject matter experts (SME). In the IIT final data collection 
phase, a revised version of a subset of the BARS developed previously was employed for 
the purpose of assessing the metrics’ utility. The research received approval from the 
Naval Postgraduate School (NPS) institutional review board (IRB). 

A. BEHAVIORALLY-ANCHORED RATING SCALES DEVELOPMENT 

The maneuver squad leader mastery model (MSLMM) developed for the United 
States Marine Corps Training and Education Command describes military decision 
making at the small-unit level and served as the foundation for developing BARS. The 
MSLMM is based on a model of small-unit leadership that describes the, “the general 
progression to mastery as a maneuver squad leader, including the progression of 
decision-making proficiency” (Ross et ah, 2012, p. 1). The MSLMM also incorporates 
the Dreyfus and Dreyfus, “five stages of skill acquisition” model of skill proficiency that 
views proficiency as progressing from novice to expert (Dreyfus & Dreyfus, 1986, p. 20). 

The MSLMM framework is premised on 15 decision-making cognitive 

competencies and cognitive and relational skills (CARS). These cognitive competencies 

and CARS were developed from surveys of non-commissioned officers (NCOs) and 

commissioned officers who had experience with MSL small-unit leadership billets (Ross 

et ah, 2012). Nine MSL performance areas were also synthesized from the NCO and 

commissioned officer surveys. “These performance areas encapsulate interactions 
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between the decision-making cognitive competencies and CARS into distinct themes” 
(see Ross et ah, 2012, p. 3). Tables 2 and 3 list the “tactical skills/tactical thinking,” and 
“communication” performance areas that were selected for their relevance to the lED- 
defeat decision-making domain (Ross et al, 2012). The selected performance areas 
described in Tables 2 and 3 provide general descriptions of the five-levels of 
competencies. They do not include specific operational training considerations. 


Table 2. MSLMM Tactical Skills/Tactical Thinking Performance Areas (From Ross 

et ah, 2012). 


Novice 

Advanced 

Beginner 

Competent 

Proficient 

Expert 

Tactical Skills/Tactical Thinkin 

> 



Understands and 

Has a better 

Confident in his 

Planning skills 

More focused on 

can employ basic 

understanding of 

decision-making 

have greatly 

taking the 

tactics with his 

the capabilities of 

abilities and 

improved. 

perspective of the 

squad. Can pick 

the enemy. Still 

now works with 

Mentally simulates 

enemy. Makes 

up cues in the 

requires some 

Higher (instead 

possible situations. 

decisions based on 

environment, but 

guidance and 

of just asking 

starts rehearsing 

the enemy’s 

has difficulty 

expects approval 

for help) to 

and considers 2'“’ 

projected actions 

understanding the 

from Higher to 

come up with 

and 3'“* order 

and abilities. Can 

meaning of cues 

make certain 

solutions. 

consequences. 

quickly identify a 

and events. Not 

decisions. Has a 

Higher seeks 

Has a better sense/ 

problem and 

knowledgeable of 

better 

him for advice. 

reading of the 

employ squads to 

how to employ and 

understanding of 

Has a better 

environment. Can 

solve it. 

use all available 

the situation/ 

understanding of 

pick up on cues 

Visualizes the big 

assets. Has 

environment and 

how to read 

and anomalies in 

picture and 

difficulty planning/ 

how to plan and 

environmental 

time to change 

considers it when 

considering a 

execute a mission. 

cues and events, 

mission. Fully 

making decisions. 

situation from start 

Considers 

and what they 

understands the 

Paints a picture for 

to finish including 

consequences 

mean to the 

implications of his 

higher and is now 

possible 

consequences. May 
make quick and 
rash decisions. 

when problem 
solving but still 
requires some time 
to make a decision. 

mission. Squad 
tactical skills are 
sharp and he 
knows how to 
employ his 
available assets. 

decision making. 

pushing 
information up 
instead of pulling. 
Can work with a 
broad intent. 
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Table 3. MSLMM Communication Performance Areas (From Ross et al., 2012). 


Novice 

Advanced 

Beginner 

Competent 

Proficient 

Expert 

Co 

mmunication 

> 



Has difficulty 

Can communicate 

Begins to adapt 

Now 

Knows how to 

gauging the 

guidance to squad, 

communication 

communicates with 

communicate with 

appropriate amount 

but it is word-for- 

styles based on 

Platoon 

Higher and squad 

of information to 

word what the 

individuals. 

Commander only 

effectively. 

report to Higher. 

Platoon 

Still requires 

for big stuff. 

Verbalizes 

Reports may have 

Commander said. 

some guidance 

including assets he 

battlefield so 

too much 

Can convey 

from the Platoon 

needs or situations 

others can see. 

information or 

knowledge to 

Commander, but 

that have strategic 

Clearly 

leave out important 

Marines. Is able to 

also pulls 

implications. Can 

communicated 

details. 

speak more 

information. 

paint a picture of 

intentions and 

Communicates 

efficiently with 

Talks to Platoon 

the situation that 

plans to squad that 

aggressively with 
squad by yelling or 
highlighting the 
negative (models a 
Drill Instructor). 

Senior people. 

Commander 
about how to 
carry out 
mission. 
Effectively gets 
thoughts out 
across the teams 
and can explain 
situations 
accurately. 

the Marines 
understand. 

they can work 
with. Feels 
comfortable 
communicating 
with Higher and 
does it with ease. 
Encourages cross 
talk. 


BARS developed for this research incorporated components of the MSLMM, 
information gleaned from a cognitive task analysis (CTA) on lED-defeat techniques 
(Ross et al, 2012), IIT operational training considerations, and SME-defined decision¬ 
making proficiency stratifications. Initially, BARS proficiency levels were designed to 
reflect the five-stages of the Dreyfus and Dreyfus skill acquisition model (Schwab et al., 
1975). However, lED-defeat behaviors for which BARS were being developed did not 
lend themselves to producing scales with five levels of proficiency. Deriving five levels 
of proficiency that could be discriminated during the fast-paced scenarios was not 
practicable. Future references to BARS will be specific to the improvised explosive 
device (lED) domain and the lED-defeat decision cycles (IE3DC) decision-making 
model explained next. 

1. lED-Defeat Decision Cycles Model 

Table 4 lists the decision-making fundamentals (objectives) in the lED-defeat 
domain (predict, prevent, detect, avoid, neutralize, and protect). These fundamentals are 
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incorporated into the lED-defeat deeision eyeles (IE3DC) model (Ross, Phillips, Moon, 
Baxter, & Cooper, 2008). “The IED3C Model was produeed as a result of this effort 
[(eoneept mapping)], in order to provide an overarehing view of the lED-defeat task from 
the perspeetive of a junior Marine, using the lED-defeat fundamentals terminology 
already in place as a construct for USMC training” (Ross et ah, 2008, p. 47). 

Table 4. Six Eundamentals of the IE3DC model (Erom Ross et ah, 2008, p. 48). 

PREDICT the actions and circumstances of the lED process. 

PREVENT lEDs from being emplaced. 

DETECT lED-related activities and lEDs before detonation. 

AVOID lEDs in order to deny the enemy a target. 

NEUTRALIZE by reducing or overcoming lEDs. 

PROTECT the force against effects of lEDs. 


As shown in Eigure 5, the IE3DC model is premised upon a eoneept map 
strueture (Ross et ah, 2008). Coneept maps are sehematic diagrams representing 
knowledge and meaning through organized eoneeptual relationships. Eundamentals of 
deeision making are depieted in the eoneept map as nodes. Eaeh node propositionally 
links one fundamental of decision making to another in an lED scenario. The concept 
map prioritizes higher-ranking fundamentals at the top of the hierarehy (protect and 
predict) over lower-ranking fundamentals at the bottom of the diagram (avoid and 
neutralize) (Ross et ah, 2008). 
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Figure 5. The Concept Map of IE3DC model (From Ross et ah, 2008, p. 49). 

In the IE3DC concept map, dependency relationships are depicted as arrows 
linking individual decision making to “processes related to one activity [having] an 
impact on another activity” (Ross et ah, 2008, p. 49). The nodal fundamentals provide 
the necessary skeletal composition of the decision-maker’s cognitive demands specific to 
the lED-defeat demands. Ten cognitive demands were initially selected for their 
relevance to the dismounted urban environment (see Table 1, in literature review 
chapter). These cognitive demands were also selected based on the potential to observe 
decision making associated behaviors without actively engaging with the trainees. 

2. Selected Cognitive Demands 

Ten selected cognitive demands that were used to develop the initial set of BARS 
were drawn from the lED-defeat domain - protect, predict, prevent, detect, avoid, and 
neutralize as delineated in the MSEMM (Ross et ah, 2008). Eor example, assessing 
normalcy in the environment is a foundational premise of the predict activity. An 
individual demonstrating mastery assessing normalcy would demonstrate exemplary 
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behavior regarding identifying anomalies in the environment (detect), and determine a 
suitable plan to mitigate the potential adverse impacts on the unit’s mission and collective 
welfare (prevent) (Ross et ah, 2008). In response, an expert in this domain would 
proactively and accurately adjust their tactics, techniques and procedures (TTPs) and 
activity, and security posture to avoid or marginalize the threat activity. An expert would 
recognize and comprehend potential second- and third-order effects of their decisions and 
base their response on the logical course of actions (COAs) from the enemy perspective 
(neutralize) (Ross et al, 2012). 

3. Assessment Factors 

a. Verbal and Non-verbal Communication 

A military leader communicates verbally and non-verbally in several ways. 
Examples of verbal and non-verbal communication include issuing orders and using hand 
signals directed at other unit members, sending radio messages to external agencies (lED 
reports to higher headquarters), and mission-related interactions with role players (key 
leader engagements). Communication mechanics and interaction techniques should be 
assessed by the unit leader’s ability to influence the target’s receptivity and comprehension 
of the message (Mueller-Hanson et ah, 2007). The format, brevity, and clarity of the 
information that is transmitted are paramount. Environmental and interpersonal 
impediments which impact communication include the emotional states of personnel, 
noise, visual/auditory range, and context. Message comprehension must be evaluated 
according to the receiver’s ability to perceive it, and interpret it. Then the message 
response needs to be in a rational and proportional manner (Mueller-Hanson et ah, 2007). 
In order to develop observable communication attributes associated with three levels of 
decision-making proficiency (novice to expert) interviews were conducted with SMEs. 

b. Tactical, Technical, and Procedural Execution 

Small-unit leaders demonstrate effective leadership by appropriately 

responding to environmental cues and ensuring actions are aligned with higher echelon’s 

intent (Yeakey, 2002). The small-unit leader’s action must not only comply with the 

constraints provided by their higher echelon (rules of engagement, ethical behavior, etc.), 
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they must cognitively align with the larger operational schemes and objectives. A Marine 
squad leader’s decision making is based on timeliness and accuracy (Goodson, McGee, & 
Cashman, 1989). It could be extrapolated that a poor decision made hastily could have 
the same adverse effects as not making a decision when conditions warrant it. Moreover, 
the BARS developed for the IIT training environment also assessed procedural 
compliance to specific safety and standardization formats (Determine How to Establish 
Security and Cordons (Detonated or non-detonated lEDs)) (Ross et ah, 2008). 

c. Location and Proximity to the Point of Friction 

The first two assessment factors, verbal and non-verbal communication, 
and TTP execution relate to the small-unit leaders’ explicit actions. These two 
assessment factors do not address the enabling decision of the small-unit leader to 
position himself within the squad so he can effectively communicate (verbal and non¬ 
verbal communication) and tactically maneuver the squad in the environment. Small-unit 
leaders are expected to: (1) have the basic ability to perceive changes in the environment, 
(2) know how to effectively communicate with their subordinates as well as the adjacent 
higher echelon, and (3) know how to employ their resources. 

A Marine squad is analogous to a system-of-systems. The unit has 
thirteen distinct elements who work in three separate, interdependent teams to achieve a 
coordinated goal. The Marine squad leader is the squad’s central decision and execution 
node through which all inputs and outputs are processed. The fire team leaders act as 
semi-autonomous responders, feeding stimuli and exercising decisions from the central 
node. Individual members of the squad act as interfaces, interacting with and 
manipulating the environment in accordance with the fire team and squad leader’s 
intentions. Execution of the decision process occurs most rapidly and correctly when the 
fire team leaders and individual squad members are in communication range and 
proximity to the Marine squad leader. 

When the squad encounters a point of friction it impacts the unit’s 
effectiveness. Without central leadership which is inherent in the Marine’s squad leader 
billet, all three separate teams can lose cohesiveness and unity. The Marine squad leader 
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focuses the fire teams and the individual members of the squad with the commander’s 
intent and purpose. When the Marine squad leader is positioned at the point of friction, 
his decision execution cycle becomes more efficient because he has first-hand knowledge 
of the situation. Conversely, the farther away the small-unit leader is from that friction 
point, the longer it will take to respond accordingly. To interrupt or attempt to influence 
the squad’s decision-making process can paralyze and potentially compromise the orderly 
function of the squad’s mission. For example, in a dismounted patrol there may be 
circumstances where the fire team elements are visually or geographically separated 
(separate alley way or road) from the Marine squad leader. The Marine squad leader may 
have difficulty effectively communicating or conducting squad-level TTPs if the 
separated elements are simultaneously engaged in kinetics actions. Additionally, the 
Marine squad leader could be right at the point of friction but encounter something 
entirely novel and may not know how to respond. 

4. Behaviorally Anchored Rating Scales Development 

Developing BARS requires a structure that allows modular adjustments to the 
metric by accounting for the variety of potential scenarios (Phillips et ah, 2006A). 
Initially where refinements and modifications were incorporated, the development 
process was envisioned to be evolutionary in nature so it would mature. The following 
are the proposed three BARS versions: post-initial research visit, post-secondary research 
visit, and post-SME interview. The final BARS would be consistent in format and 
content with the post-SME interview version. 

a. Initial Research Visit to the Infantry Immersion Trainer 

An initial visit to the IIT was conducted in order to observe and become 
informed about the IIT training and operations. The IIT staff provided an operational 
overview describing the current IIT training, assessment methodology, technology, and 
training participants. Observations of Marine small-units conducting operational training 
with semi-structured scenarios were conducted in both indoor and outdoor training 
ranges. A brief was provided to the IIT staff on this thesis research background and 
purpose, and addressed how the BARS would provide a more objective and quantifiable 
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assessment of trainees’ proficiency. The majority of the feedback from the IIT staff 
indicated that any assessment techniques would have to be easy to use and quick to 
implement. Moreover, any assessment would need to have a negligible impact on the 
trainer’s attention and interactions with the training audience. 

IIT operational training is designed and developed around a particular 
operational scenario and/or set of mission essential tasks. IIT scenario development is 
typically illustrated through a storyboard. This storyboard format facilitates the IIT staff 
in organizing the scenario execution and narrative. A storyboard delineates pivotal 
milestones within the scenario that a Marine small-unit will encounter. As depicted in 
Figure 6, the information at the top of the scenario storyboard describes the unit (1/7), 
training day (TD-1), training facility (indoor), scenario number (1), and description (QRF 
(quick reaction force) to ANSF (Afghan national security forces) lED Find at the Sqd 
(squad) level). Along the left-hand column tactical objectives, atmospherics, visuals 
(pyrotechnics), and disposition/composition of the enemy role-players (enemy activity) 
are described. A graphical depiction of the expected scenario execution is also presented. 
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Mission; Link up with ANSF 
patrol IVO of the market 
lOT assist them in securing 
the area around an lED 
that they have identified. 

Atmospherics : ANSF 
having trouble keeping 
locals dear of lED. 

Visuals : lED covered with 
disturbed earth in the 
market. 

-Pressure plates IVO lA 
and 3B lEDs. 

Enemy Activity: 

-lA and/or 3B lEDs 
detonate if pressure plates 
are stepped on. 


1/7 TD-1, Indoor Scenario 1: QRF To ANSF lED Find (Sqd Level) 



Figure 6. IIT Scenario 1 Storyboard: QRF to ANSF lED Find (From Fennell, 2013B). 
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The IIT scenario storyboard is an internally distributed planning and 
operations document. The IIT staff can change the specifics of the scenario prior to and 
during scenario execution to meet the training requirements of a particular unit. The 
training staff will only disclose general scenario atmospherics and tactical objective(s) to 
the training participants. The IIT staff adjusts scenario dynamics and atmospherics as 
needed to adapt to the squad’s level of proficiency and to achieve specific learning 
objectives. 


b. Behaviorally Anchored Rating Scales Initial Development 

The goal for this phase of the research was to synthesize the existing body 
of knowledge on BARS development and develop an initial set of BARS. Table 5 lists 
and describes the ten cognitive demands selected from the IE3DC model, and provides 
descriptions of lED related behavioral indicators, conditions, and cues. 
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Table 5. IE3DC Model Cognitive Demands and Descriptions 
(From Ross et al., 2008, pp. 51-56). 


Cognitive Demands 

Descriptions 

1. Assess Normalcy in the 
Environment 

Establishing a baseline understanding of the environment is one 
of the pillars on which all lED-defeat decisions rest. 
Knowledge of what is normal - from behaviors of the populace 
to placement of specific objects in the operating environment - 
enables detection of change and identification of anomalies that 
may be indicative of insurgent signatures and devices in the 
process of emplacement. Normalcy may be passed from unit to 
unit during Reliefs in Place, from leadership to platoon during 
battle space preparation, and from Marine to Marine during the 
course of patrols and convoys. 

2. Assess Probable lED 
Locations 

In addition to assessing when lED attacks are likely. Marines 
continuously assess where attacks may occur. These 

assessments occur at a general level - i.e., knowing insurgent 
operation areas and trends in behavior - and at a specific level 
- i.e., knowing hot spots and evaluating the features of the 
environment that favor enemy activities. In conjunction with 
assessing where an lED may be located, another component of 
this judgment is where particular types of lEDs are likely to be 
used. Eor example, high-yield lEDs such as [explosively 
formed penetrators] EEPs are more typically used on rural 
routes, while lower-yield lEDs can be anticipated more 
frequently on urban routes. Marines’ assessments of probable 
lED locations have a direct impact on route planning activities 
as well as on-the-fly decisions (i.e., during a patrol or convoy) 
regarding which routes to traverse. 

3. Notice Indicators of a 
Dangerous Situation 
(Terrain or People) 

This decision reflects how skilled Marines detect an lED threat 
by focusing not just on indicators of a device, but also on 
indicators throughout the environment that suggest they may be 
entering a zone where an lED is located. This decision 
necessarily interacts with those under Predict; Marines use 
expectations they have about where lEDs might be located and 
when they are most likely to be emplaced in order to prime 
their recognition of danger indicators. There are two primary 
indicators Marines assess when noticing a dangerous situation - 
terrain, and people and their activities. Terrain features 
include: cues on the road - e.g., potholes; tread patterns - and 
cues around the road - e.g., structures such as bridges and 
abutments. People and activity features include: the extent of 
current and recent traffic; the absence of civilians and children; 
and movement in buildings. 


Continued on next page 
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Continued from previous page 


4. Notice Indicators of an 
lED Device 

Another pillar of lED defeat lies in noticing the indicators of a 
device. This includes noticing components of a device in the 
context of vehicle and home searches, and identifying lEDs 
that are emplaced hy their component parts. Exhaustion and 
complacency are common challenges associated with 
identifying lEDs in the environment. However, when a 
Marine’s attention is focused he can anticipate an lED attack, 
or when his attention is directed to high-threat areas as a result 
of his experience, detection becomes more manageable. An 
extensive list of cues and factors involved in this judgment are 
provided in the cognitive demand tables (CDTs). As the 
USMC well knows, these cues and factors are only as current 
as the improvisations of the enemy. 

5. Assess the Nature and 
Extent of All Current 
Threats 

This decision concerns the devices as well as other insurgent 
activities that surround them. Marines must simultaneously 
judge what they have detected and what threat it poses, and 
what other threats may be posed by other devices or forms of 
attack. These judgments consider whether devices may be 
hoaxes, whether secondary devices may exist, and the 
possibility of ambush - any of which is possible in any 
situation. Marines consider information available about the 
immediate scene, the protective equipment they have in place, 
their vulnerability, and even the nature of reporting surrounding 
a device. 

6. Determine How to 

Trace to the Point of 
Origin 

Eor devices that suggest a command wire-initiated device. 
Marines may attempt to trace elements of the device to the 
Point of Origin, or POO, site. Cues considered are wires and 
other elements. Tracing is made challenging by the potential 
for booby traps and secondary devices, and because some POO 
sites may be great distances from the point of detonation. 
Marines must judge the risk associated with tracing the wire. 
When risk is deemed high. Marines may visually attempt to 
trace a site, and/or walk identifiable paths to the POO site, a 
strategy that amplifies the challenges. 

7. Identify and Respond 
to Triggerman 
(Detonated or non- 
detonated lEDs) 

Eor devices that require command detonation. Marines look for 
triggermen. The search may include looking for a line of sight 
and places that are concealing, and typically goes hand-in-hand 
with searching for the Point of Origin site. In cases where a 
triggerman is identified. Marines must then consider how to 
deal with him. Options here include giving chase, firing, or 
doing nothing, and each of these options may be considered 
with regard to safety, feasibility of success, and security. 


Continued on next page 
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Continued from previous page 


8. Determine How To 
Establish Security and 
Cordons (Detonated or 
non-detonated lEDs 

The goal of this decision is one of safety and of maintaining the 
integrity of the environment around a detected lED. Marines 
want to keep everything within the cordon that was there at 
detection, and keep anything else out. In determining when to 
establish a cordon, Marines are challenged hy the often lengthy 
duration of time EOD [(explosive ordinance disposal)] may take 
to arrive on scene. In determining how to establish a cordon. 
Marines must take into account the terrain around the lED site, 
manpower, and vehicles at their disposal. 

9. Consider How to 
Respond Immediately 
to Detonation 
(Detonated lEDs) 

Upon detonation. Marines must make split-second decisions 
about how to respond. These decisions include whether to 
approach the detonation site, whether to dismount, and where 
to place vehicles in a convoy. All of the other Neutralize 
decisions and judgments above come immediately into play, 
and are made all the more challenging by the stress of enduring 
the detonation and deep-seated first impulses to help victims 
and fight. Platoon leadership must also consider when and how 
to communicate with higher headquarters. Other post¬ 

detonation decisions include assessing the damage, responding 
to injuries or casualties, and responding to disabled vehicles. 
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The initial version of the BARS was based on the ten cognitive demands 
listed and described in Table 1 of the literature review chapter. As depicted in Figure 7, 
the BARS format consisted of three parts. Each part was annotated by the part number (I, 
II, and III) in bold red lettering. Part I stated the general scenario description 
(unexploded lED), and the associated cognitive decision related to the perceptual task 
(notice indicators of an lED device). In addition to the cognitive decision, part I also 
specified the perceptual task related to the lED (identify lEDs). Part II is a 5-point scale 
based on the Dreyfus and Dreyfus skill acquisition model where proficiency ranges from 
novice to expert (Dreyfus & Dreyfus, 1986). The scales were intended to be used by the 
assessor to observe the Marine squad leader’s behavior, and then mark the proficiency 
level associated with the cognitive decision in part I, using the related descriptors of 
observable behaviors annotated in Part III. Part III consisted of multiple behavior factors 
(listed across the rows), containing five levels of observable cognitive linked behaviors 
(columns). 

The central tenet of the BARS utility was the ability to associate 
observable behaviors demonstrated by the training participants with objective criterion 
that differentiated between highly proficient (expert) and novice behavior. In addition a 
report on CTA was used to develop initial descriptors (Ross et al, 2012). However, 
observable indicators of the decision-making behavior were not found in the report due to 
an incompatibility between the scope of the CTA on lED-defeat performance and the IIT- 
specific training. The existing literature provided information to develop the set of 
behavioral cues for the BARS. As depicted in Eigure 7, the cognitive demand (notice 
indicators of an lED device) includes observable decision-making behaviors (identifiable 
cues of lED danger areas, lED assessment techniques, and cultural and environmental 
lED emplacement knowledge). During the researcher’s second visit to the IIT an initial 
version of BARS were employed to test the BARS utility. 
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Scenario: Unexploded lED 


Decision: Notice Indicators of an lED Device (Identify lEDs) 
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Figure 7. Example of an Initial Behaviorally-Anchored Rating Scale: Notice Indicators 

of an lED Device. 


c. Second Research Visit to Infantry Immersion Trainer 

The purpose of the second visit to the IIT was to further develop the 
researcher’s familiarity regarding IIT operations, training and scenario development, and 
BARS utilization. The first day focused on familiarization with IIT operations, and 
scenario development and execution. During the following three training observation 
days, the focus was on acclimatizing to the IIT training environment and utilization of the 
metrics already developed. The following list includes the research objectives for the 
second visit: (1) To achieve a more comprehensive understanding of the IIT staff roles 
and responsibilities; (2) better understand the IIT training scenario development and 
execution process; (3) comprehend the limitations and constraints of the IIT staff and 
facilities; (4) elicit feedback from the IIT staff regarding thesis research and BARS 
development; and (5) coordinate follow-on final data collection. 


48 









The following observations occurred over a four-day period. Each day 
provided additional insights and lessons learned. 

(1) Day One Observations. Upon arrival at the MCBCP IIT, the 
IIT director and operations officer provided an overview of IIT operations ranging from 
staffing and training issues, to facilities and contractor support considerations. 
Additionally, information was provided regarding first Marine Expeditionary Eorce (I 
MEE) IIT utilization policy, and ongoing experimentation and assessment projects. In 
subsequent conversations, video and audio capturing technologies were discussed as a 
means to facilitate after-action reviews (AARs) and trainer assessments. 

Between the first two IIT visits, a change occurred in the manner 
in which the IIT staff captured and integrated selected scenario video segments into the 
AAR. The legacy instrumented video/audio capturing system called TVCS (tactical 
video capture system), had an inherent and significant delay in processing selected videos 
for the AAR. Given that training scenarios typically required 30-60 minutes to complete, 
time spent processing TVCS videos for the AAR had become too time-consuming and 
ineffective as the primary debriefing aid. The IIT staff implemented a change from the 
instrumented close-circuited TVCS to a hand-held contour camera system employed by 
individual IIT staff. Usage of handheld cameras reduced the inherent processing delay in 
the TCVS/Spotlight systems from 20 minutes to 5 minutes. This improvement allowed 
the IIT staff to increase the number of training scenario runs significantly over a four-day 
training evolution. 

In additional conversations, the IIT operations officer elaborated 
on their training philosophy regarding scenario development, where he emphasized a 
winnable, applicable, and coachable approach to training Marines. Each event must 
afford training participants a modicum of mission success and personal growth. Salient 
tactical and technical execution points must demonstrate relevance and applicability to 
preparing the training audience for future operational deployments. Einally, events must 
emphasize an open-ended decision-making process, that is to say, there is not always one 
correct way to respond to a given situation. The training environment must be conducive 

to and provide the impetus for IIT staff guidance and COA training audiences. 
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An example of the BARS was provided to the IIT operations 
officer and he was asked to provide feedback regarding the applicability of this metric. 
His initial feedback focused on the individual units and their respective pre-deployment 
training program (FTP) cycle. He stated that units further along in their PTP cycle have a 
distinct and expected advantage regarding TTP proficiency and execution compared to 
units who are not as far along in their own PTP cycle. Units early in their respective PTP 
cycles are adversely affected by personnel turnover, unit cohesion, and leadership 
continuity issues. His opinion was that a trainer who uses BARS would need to consider 
where the unit is in their PTP when assessing of the unit’s performance. 

(2) Day Two Observations. Day two consisted of observing 
squads engaging in scenario runs in the indoor facility. Figure 8 shows the AAR Room 
which served as the main debriefing location for the indoor scenario runs. A total of six 
training scenario runs and/or AARs were observed, specifically training day one, indoor 
scenario one and three. 

After querying the unit’s Platoon Commander regarding scenario 
training run information, the Platoon Commander confirmed that the squad was in a QRF 
role providing assistance to the ANSF for securing the bazaar due to an identified lED 
threat (training day one, indoor scenario one). With this knowledge, it was decided that 
the following three BARS would be used: “Notice indicators of an lED device,” 
“Determine how to establish security and cordon,” and “Consider how to respond 
immediately to detonation.” The following paragraphs summarize observations and 
highlights of day two training scenario runs utilizing the BARS. 
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Figure 8. Phase I AAR Room (From Thelen, 2013). 


(3) Training Day One, Indoor Scenario One. 1st Squad: As 
depicted in Figure 6, the Marine squad’s primary mission objective was to interact with 
the ANSF at a simulated bazaar and support a cordon and security in the bazaar around 
an identified lED. A secondary objective was to support a simulated EOD unit with 
protection and security. Figure 9 shows the Marine squad entering the indoor training 
facility and moving slowly toward the bazaar (objective area). 
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Figure 9. Marines navigate through alleyways of the Phase I of IIT 

(From Thelen, 2013). 


The lead fire team element of the squad was spearheaded by a 
Marine holding a compaet metal deteetor (CMD). The lead Marine, referred to as the 
Sweeper, carried and used the CMD to search for high-density objects, such as mines or 
lEDs in the intended navigable direction and path of the squad’s movement. Once the 
CMD detected a metallic (high-density) object under its sensor, it emitted a visual and 
auditory indication. As previously discussed, the IIT is a converted former tomato 
packing plant with some legacy infrastructure artifacts, such as metal rebar throughout 
the concrete surface of the indoor facility. This particular artifact caused the CMD 
device to continually emit indications of highly-dense objects. The IIT staff 
accompanying the respective unit compensated for the metal artifact by providing the 
Sweeper an audible indication of the correct emanation. A verbal, “soft,” “weak,” 
“hard,” or “strong,” indication was given by the trainer to simulate the level of detection. 
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As the squad approached and entered the objective area, the 
Marine squad leader moved forward and interacted with the ANSF unit leader. It 
appeared that the squad leader was receiving a brief from the ANSF unit leader regarding 
the bazaar danger area. Additionally, the ANSF unit leader signaled the general 
proximity of the lED device at 3B (Figure 10). With the general proximity of the lED 
shown, the Marine squad leader paused a moment and then visually confirmed the ANSE 
leader’s claim. The “Notice indicators of an lED device” BARS (see Eigure 7) was 
utilized at this juncture in the scenario. 

The “Notice indicators of an lED device” BARS consists of three 
factors shown in Eigures 10, 11, and 12. These three factors include noticing danger area 
cues, lED cue detection response performance, and cultural and environmental 
discrimination of lED emplacement characteristics. The first behavior factor (Eigure 6), 
“Notice lED danger area cues” was assessed, followed by the other two behavior factors 
in order (Eigure 11 and 12). When observing the Marine squad leader’s response to the 
ANSE unit leader’s hand-and-arm signals and his subsequent scrutinizing the terrain, it 
appeared that the Marine squad leader was attempting to perceive and categorize the 
lED/mine cues from the terrain. After reading the descriptors associated with each point 
on the BARS scale, the rater determined that the fourth marker, “understands cues that 
indicate danger areas with lEDs,” was an appropriate rating. 

In highlight, assessment of the Marine squad leader’s behavior 
related to identifying lED danger area cues was completely subjective in nature. This 
assessment used length of time spent visually looking into the danger area as a positive 
indication of a systematic search methodology. Additionally, the assessment assumed the 
amount of time the Marine squad leader spent surveying the danger area was positively 
correlated to correctly identifying the lED/mine location. These two assumptions were 
based on three unsubstantiated biases that influenced the assessment. Eirst, the Marine 
squad leader was completely receptive to the ANSE unit leader. This was unfounded 
because there was not the ability to objectively verify the Marine squad leader’s 
confidence in the ANSE unit leader’s knowledge. Second, the Marine squad leader had 
competent expertise to locate and identify the lED/mine threat at 3B (remote control, 
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command-weight switch, command-wire, etc.). This information about the Marine squad 
leader was unknowable without retrieving his demographic information. Third, the 
Marine squad leader comprehended the approximate lEDs/mine’s effective casualty 
radius (ECR). The Marine squad leader’s knowledge and awareness of the lED 
characteristics was also unknowable without demographic information on the Marine 
squad leader. These assumptions and their underlying biases were not supported or 
substantiated by other objective evidence. 
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danger areas w/ 
lEDs 


□ 


Associates cues 
that Indicate 
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Eigure 10. Behaviorally-Anchored Rating Scale Version One: Notice Indicators of an 

lED Device (LED Danger Area Cues). 


The Marine squad leader’s verbal and non-verbal (hand-and-arm 
signals) interactions with other members in his unit were observed in addition to his 
interactions with the ANSE unit leader. It is noteworthy that the ability to hear the 
Marine squad leader’s radio communication from the Phase I AAR Room was extremely 
difficult given the limiting acoustics of the indoor facility, and limited access to the 
training participant’s radios. The indoor closed-circuit television (CCTV) camera 
perspective and AAR room monitor resolution additionally prevented the rater’s ability to 
visually decipher verbal interactions between the Marines, role-players, and IIT staff 
during training scenario runs. Communication behavior of the Marine squad leader could 
only be inferred through observing his non-verbal communication. After observing the 
Marine squad leader’s communicative behavior, it was ascertained that the Marine squad 
leader’s “lED danger area response performance” was consistent with BARS level four 
“competent real-time identifying of danger areas with lEDs” (Eigure 11). 

In hindsight, the assessment of this behavior factor had similar 
pitfalls described in the previous “noticing indicators of an lED device” factor. There 
was not a distinct means of articulating the timing and/or accuracy of this behavior 
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factor’s performance criteria. Additionally, it was not possible to observe when the 
Marine squad leader utilized a reference such as a checklist from the Phase I AAR Room. 
Having distinct and definable response performance criteria codified prior to scenario 
training run execution would enhance this factor’s usability. 


Req verbatim use Uses, not req, lED Competent but Competent real- 

|~| lED checklist to chkiist to identify delayed w/ time identifying 

identify danger danger areas w/ identifying danger danger areas w/ 

areas w/IEDs lEDs areas w/lEDs w/o lEDs 

lED checklist 

Figure 11. BARS Version One: Notice Indicators of an lED Device (lED Danger Area 

Response Performance). 

The Marine squad leader was observed surveying the bazaar 
danger area for potential lEDs. It was inferred that the Marine squad leader was actually 
searching for physical indicators of lED emplacement. lED emplacement knowledge 
was critical given that lED emplacement characteristics can change substantially from 
one geographic region to another. Understanding the geographically specific lED 
emplacement characteristics of the deployable operational environment provides a greater 
situational awareness and improves self-protection. Although there are distinct 
differences in lEDs used in different locations, there are specific lED heuristics shared 
across these geographic domains. Euture reference to lEDs will be specific to ground- 
borne, stationary devices. 

How the lED is actuated provides important information related to 
the lEDs physical and emplacement characteristics. The IIT trainers and staff provide 
limited information to the training participants regarding the simulated lEDs effective 
causality ranges (ECRs) in the IIT training environment. The Marine squad leader’s 
ability to recognize lED emplacement characteristics is important in determining standoff 
ranges and cordon considerations. When rating the Marine squad leader’s lED 
emplacement knowledge and awareness using the BARS in Eigure 12, he was rated a 
“competent” evaluation because no Marine stepped on or actuated the simulated mine in 
the scenario. 
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Figure 12. BARS Version One: Notice Indicators of an lED Device (LED Emplacement 

Knowledge and Awareness). 

As the cordon was quickly established around the bazaar area with 
the assistance of the ANSE, it appeared that the ANSE were tasked to move the civilians 
(role-players) out of the bazaar area cordon. The size of the simulated bazaar was 
distinctly smaller than what is available in the outdoor (Phase II) training range. It 
appeared that it would now be appropriate to utilize the “determine how to establish 
security and cordon” BARS. Eigure 13 shows this BARS has four behavior factors 
related to locating the lED. They are as follows: lED location, lED emplacement 
knowledge and awareness, 5/25 meter searches, and integrity and security of the cordon 
until EOD arrives. After establishing the cordon at the entrances to the bazaar, the 
Sweeper was tasked to clear designated locations of the cordon area. 

This metric’s first two behavior factors were previously discussed 
in the “notice indicators of an lED device” BARS and assessed at the same proficiency 
levels. The drawback associated with nesting several distinct and independent factors 
was a significant lesson learned in the BARS development process. The next behavior 
factor relating to the 5/25 meter searches of the danger area was done independent of the 
Marine squad leader’s direction. Observing whether the Marine Squad leader 
individually conducted 5/25 meter searches was impossible from the vantage point of the 
Phase I AAR Room. The only indication of this behavior’s proficiency was the number 
of potential lEDs discovered and false positives. Since this information was not 
technically known prior to scenario execution, nor incorporated into the existing metric, 
this behavior factor was not assessed. 
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Scenario; Unexploded lED 


Decision; Determine How to Establish Security and Cordons 
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Figure 13. BARS Version One: Determine How to Establish Security and Cordons. 


The squad’s Platoon Commander was also observing the scenario 
execution from the Phase I AAR Room and was role-playing the simulated combat 
operations center (COC) (squad’s higher headquarters). The COC was not receiving 
radio communication from the Marine squad leader regarding the identified lED. The 
Platoon Commander commented aloud that the lack of radio communication in the form 
of an lED 9-Line report was not normal. This radio communication irregularity was an 
issue that in hindsight plagued future training scenario runs. The issue was identified and 
attributed to two factors. The first factor related to an improper radio crypto logic fill, 
and the second factor related to the internal infrastructure of the Phase I facility. Both 
issues separately could cause a severe impact to sustaining communication and both 
factors together would make radio communication nearly impossible. 

Radio communication was successfully established after the 
Platoon Commander installed the proper radio crypto logic fill and adjusted his 
transmission location and antenna configuration. When the Marine squad leader 

57 




successfully communicated the lED 9-line report it was in an incorrect format and the 
report’s contents were not relevant to the IIT scenario. The Marine squad leader did not 
provide the IIT specific building location and the radio communication cadence was 
erratic. This issue forced the Platoon Commander to explain over the radio the correct 
reporting format, creating friction and delay to the unfolding scenario. Additionally, the 
COC had to direct the Marine squad leader to search and report the status of other 
potential lEDs in the cordon area. In hindsight, these types of issues occurred repeatedly 
throughout the training day, and were not mutually exclusive to this squad. Einally, the 
simulated EOD unit (IIT staff) was brought into the scenario, the EOD disarmed the lED 
and the scenario came to an end. 

The final BARS behavior factor, “integrity and security of the 
cordon until EOD arrives” was assessed. As depicted in Eigure 13, the behaviors 
included in this behavior factor were the exclusive control of the cordon with respect to 
avenues of approach, the internal quarantine of lED/mine devices (location, 
identification, and marking), and finally the EOD link-up plan and execution. The 
scenario did possess some artificiality with respect to the trainer’s compression of the 
EOD timeline. The amount of time required by the simulated EOD to arrive, get briefed 
and finally neutralize the known lED was relatively fast (<30 minutes). Conversely, the 
time required in an operational environment for EOD to arrive after receiving the request, 
receiving a proper briefing, and then disarm the unexploded ordinance (UXO) would 
requires substantially more time than what was available in the scenario timeline (indoor 
scenarios usually last approximately 45 minutes to one hour). This artificial time 
consideration was not meant to question the relevance of the cordon actions, rather the 
truncated time line forced the Marine squad leader to ensure his decisions were made in 
an efficient manner. Observing the EOD/Marine squad leader’s interaction over a ten 
minute period, the rater assessed that he competently executed the behaviors at level 
three. 

With the lED/mine disarmed, the scenario training was complete 
and the squad moved to the Phase I AAR Room. The trainer began the AAR by 
explaining the video’s perspective shown on the monitor would be from the point of view 
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of a potential adversary or threat. The trainer emphasized that the squad should 
comprehend the enemy’s perspective in dismounted environments. He continued by 
stating the Marines needed to appreciate their adversaries as active components of the 
battle space. The trainer subsequently presented video excerpts of the squad as they 
progressed through the scenario training run. Each pause of the video focused on 
individual and collective deviations (learning points) related to internal/external 
communication, roles and responsibilities, TTP conduct and execution, and 
cordon/security conduct and procedures. The trainer’s point was to elicit individual 
perspectives of the situation, present the actual ground truth of the event, and contrast the 
training participant’s perspectives against the actual transpired events. 

Training Day One, Indoor Scenario One—I"** Squad: The 

second scenario run was significantly different from the first scenario run due to a Marine 
becoming a casualty as a result of stepping on a simulated pressure-plate mine between 
A5 and A3 (see Figure 6). Two casualties occurred from the detonation in this scenario, 
a Marine and an ANSF soldier. Both casualties were dragged into the mock police 
station compound located at A5. The Marine squad and the ANSF fire team completely 
collapsed into the A5 compound and surrounding exterior wall. 

After observing the post-detonation environment, it appeared 
appropriate to utilize the “consider how to respond immediately to detonation” metric. 
As depicted in Figure 14, there were four behavior factors related to this BARS: 
awareness of personnel location, 5/25 meter searches, establishment and maintaining of a 
cordon, and reporting considerations to higher headquarters. The transpired events gave 
the impression that the metrics would be difficult to utilize. Three factors made assessing 
the Marine Squad leader’s behavior using the BARS difficult. First, the affected 
casualties required immediate medical attention so they were quickly moved from post¬ 
detonation into A5 compound without direct involvement of the Marine squad leader. 
The inability to ascertain the Marine squad leader’s verbal commands and queries 
prevented utilizing the “awareness of personnel location” behavior factor. Second, when 
the squad collapsed their positions into the A5 compound, it was done in such an 
expeditious manner, that it prevented observing whether or not the squad conducted a 
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5/25 meter search for secondary or tertiary lEDs or mines. Third, establishing and 
sustaining the cordon and security should be a separate and independent BARS. The 
remaining behavior factor, ‘reporting considerations to higher headquarters’ was 
addressed later in the scenario training run. 


Scenario: Exploded lED 

Decision: Consider How to Respond Immediately to Detonation 
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Figure 14. BARS Version One: Consider how to Respond Immediately to Detonation. 


With the Marine and ANSF units moved inside the A5 building 
compound, it was not possible to hear radio communication from the Platoon 
Commander’s radio in the Phase I AAR Room. It became apparent that the 
communication issue presented itself again and the Marine squad leader and the Platoon 
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Commander were observed trying to establish 2-way communication. This lack of 
communication spanned more than five minutes and the delay adversely affected rating 
the Marine squad leader’s performance. With this communication delay, it became 
apparent the inclusion of a time dimension in any future BARS would not be appropriate 
given the difficulties in establishing and maintaining two-way radio communication. The 
BARS for “Consider how to Respond Immediately to Detonation” was deemed 
ineffective for assessing the Marine squad leader’s decision-making performance for this 
particular cognitive demand. 

Once the Platoon Commander (COC) changed his location, 2-way 
radio communication was established. During the radio transmission from the Marine 
squad leader, the COC heard the same improper procedural radio format (lED/UXO 9- 
Line report). This made the Platoon Commander rhetorically comment about there was 
additional training that was needed regarding proper radio procedures and format. In 
subsequent two-way radio communication the Marine squad leader demonstrated 
difficulty in medically classifying the two casualties. After the radio exchange, the 
Platoon Commander commented again that additional training was necessary in medical 
classifications and reporting procedures. Finally, the Platoon Commander indicated that 
there was not a Marine in the Sweeper role after the original Sweeper had become a 
casualty. The Marine squad leader needed to task another marine to that role. Without a 
Marine employed in the Sweeper role, the squad had a very remote chance of 
substantially preventing additional lED/Mine events. The Marines navigated back to the 
entrance of the Phase I facility, and the scenario training run culminated. 

When the squad established itself in the Phase I AAR Room, the 
trainer began the AAR by covering the events that transpired up to the Sweeper stepping 
on the mine. The trainer elaborated on the roles and responsibilities of the Sweeper and 
the Marines around him. lED marking schemes and techniques were addressed, 
specifically shaving cream or baby powder as suitable means to mark lEDs and cleared 
areas. Additionally, the trainer elaborated on the squad’s lack of explicit and/or vocal 
lateral communication. The trainer expressed that only through vocal and explicit 
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communication can their individual intentions and movements build a common 
situational picture for the other unit members. 

Finally, the trainer paused the video where the Sweeper actuates 
the mine. The video shows the Marine in the Sweeper role visually recognizing the 
presence of the disturbed earth, but remains silent. The trainer emphatically proclaimed 
that each Marine had an obligation and responsibility to individually scan the area around 
them and alert the squad if there were any indications of potential lEDs/mines. The 
trainer concluded by saying that it was imperative for everyone’s survival that when there 
are confirmed lEDs in the local area everyone should remain vigilant in scanning and 
ensuring there are not secondary and tertiary devices. 

Training Day One, Indoor Scenario One—Squad: Due the 

length of the previous AAR, observation of this squad’s scenario run was impossible. 
The 3 Squad had completed their scenario training run while the previous scenario run 
debrief was taking place. In hindsight, the long occupation of the AAR room not only 
prevented the observation of this scenario run, but continued long debriefs prevented 
subsequent observations as well. Moving to another location, such as the Phase I video 
operator’s room located between C2 and C4 (labeled “out of play” on Eigure 4) could 
mitigate this issue. 

Interview with IIT Head Trainer 

After explaining the purpose and intent of the thesis research, the 
IITs head trainer was asked to provide an assessment of the research BARS potential 
utility. He responded that BARS would have to be strictly an enabler to the trainer’s 
assessment. The head trainer also commented that any BARS should not be tied to the 
promotion or demotion of the unit leader under training. He stressed that units under 
training have different challenges depending on where they are in their PTP cycle. He 
explained that it would be unfair to compare a unit’s leadership performance against units 
that are further along in their PTP cycle. Sample BARS were provided for him to review 
and provide feedback. 
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(4) Day Three Observations. The previous training day foeused 
exelusively on squad-level seenario runs in the indoor (Phase I) faeility. Day three 
observations foeused exclusively on the outdoor (Phase II) training range located 
adjacent to the Phase I facility. Figure 15 shows the Phase II training range is divided 
into northern and southern lanes where platoon- and company-level units conduct 
scenario runs. Phase II observations were exclusively restricted to the TVCS operator’s 
room (Figure 16), located in the IIT administration building (Figure 15). 
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Figure 16. Phase II Tactical Video Capture System Operator’s Room 

(From Thelen, 2013). 

During the initial orientation to the TVCS, it became apparent that 
any observation would have to be more deliberate and focused compared to Phase I 
observations. The TVCS is comprised of a large-scale CCTV camera network which is 
remotely-controlled and managed by TVCS operators. The TVCS operator’s console and 
monitor (see Figure 16) was comprised of several cameras vantages of the outdoor training 
range. Numerous outdoor cameras were selected on the TVCS operator console and 
monitor and prevented a singular perspective of the scenario run. Interactions with TCVS 
operators were limited as to not impede their activities and responsibilities. As a result of 
this constraint, there was limited opportunity to control the camera perspective and scene 
selection. To mitigate this constraint, the TVCS operators agreed to verbalize the camera 
perspectives. Additionally, the TVCS operators agreed to discuss where the individual 
squads or fire teams were located throughout Phase II training ranges. The two observed 
scenario runs used the same platoon QRF scenario storyboard (see Figure 17). 
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Enemy Situation: 3-6 man enemy 
element is engaging an ANSF unit 
conducting a cordon and search IVO 
116 with smai arms fire. The ANSf 
unit is being engaged from the multiple 
positions to the west, north, and east 
of the OBJ and have sustained at least 
(1) friendly casualty. 

Mission: Provide QRF support to ANSF 
unit in a SAF engagement in the 
northeast corner of the vilage lOT 
eBminate all enemy positions and 
assist them in completing their cordon 
and search IVO 116. 

Atmospherics: Most civiBans are taking 
cover indoors due to firefight. 

-.A group of civiBans are interfering 
with cordon positions because of 
civiBan injuries sustained in firefight. 

Visuals: 

-3 ANSF teams (5 soldiers each) IVO 
L5, inside 18, and IVOI16 engaged with 
enemy. 

-HME in 116. 

-(!} unarmed HME manufacturer in 
116. 

-(2) Wounded ANSF soldiers on the 
south side of 116. 

-CiviBans yeling to ANSF at the OBJ for 
assistance to treat civiBan injuries. 
-Injured civiBans in H4 courtyard. 

Enemy Activity: 

(1) Shooter engaging ANSf teams 
WO L5 and in 18. 

(1) Shooter engaging ANSF team WO 
116 from the 2"** story of K2. 

(1) Shooter engaging ANSF team WO 
116 from the L15-L16 courtyard. 
(1) Shooter on standby WO H18. 

(1) Shooter on standby WO F18. 

(1) Shooter on standby WO FI. 


3/7 T-DAY 3 Scenario: PLT QRF 



Figure 17. IIT Phase II - Training Day 2, Scenario: Platoon QRF Storyboard (From Fennell, 2013A). 
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Outdoor Scenario Day Three—3/7 Platoon: As the Marine 
platoon entered the outdoor facility, the platoon split itself into three squad elements. 
Each element independently entered through different entrances along the northeastern 
and western entrances of the Phase II north lane (see Figure 17). Additionally, Marine 
role-players playing adversaries or ‘Shooters’ were discharging their weapons as a means 
to simulate atmospherics of the scenario. Civilian and ANSF role-players in the vicinity 
of K3 compound were observed requesting the platoon’s assistance. The three squads 
were observed navigating toward the ANSF and civilian role-players. As the squads 
approached the vicinity of the “K” sector of the Northern training range (see Figure 17, 
bold black letter) they began to receive small-arms fire (SAF) from different Shooters in 
the K2, F15/16, and Hl/fI2 buildings. Squads subsequently cleared K2 and F15/16 
buildings of Shooters, but the H1/112 Shooter continued to engage a squad in the vicinity 
of K3 compound. Finally, the squads subdued the Shooter in the vicinity of H1/H2 
buildings, and then all three squads exfiltrated (EXFIFF) through the northern exits of the 
Phase II training range. 

During the platoon’s engagement in the scenario, there was 
considerable difficulty using the BARS. The inability to see and/or hear the Marine 
platoon or squad leader’s individual actions and/or radio communications impeded using 
the BARS. A critical difference between observing scenario runs in Phase I and II was 
the scale of the two respective facilities, and the ability to focus only on one squad vice 
three. The scope of indoor facility presented a significant advantage compared to the 
outdoor facility for the purposes of this research. As a result, the potential exclusion of 
the outdoor training range for the BARS development was reevaluated at the conclusion 
of the week’s training evolution. Observation of the following platoon’s outdoor 
scenario run reinforced the notion of how difficult rating the Marine squad leader’s 
decision making performance would be in the outdoor training range. Due to the lack of 
control in manipulating, selecting and focusing the cameras’ point of view and display on 
the TVCS monitor and console, inclusion of the outdoor facility for this research was 
deemed not suitable. 
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d. 


Lessons Learned from Second Visit to IIT 


Lessons learned from the second IIT visit, proved invaluable to improving 
the scope, format, and content of the subsequent version of the BARS. Ten BARS 
proved to be too cumbersome to use in a 30-40 minute fast-paced scenario. The intent of 
future data collection opportunities would be to decrease the number of BARS used. 
Additionally, a modification of the format and content of the BARS metric with respect 
to Part I, II and III was necessary. 

As depicted in Figure 18, the BARS Part I was made more explicit with 
respect to the cognitive demand (in accordance with the MSLMM). Additionally, the 
Part I observable decisions were tailored to a specific location within the scenario 
training run (see Figure 19). The decision to choose a single scenario run rather than 
multiple scenario runs was due to time and utilization considerations (All squads would 
use training day one, indoor scenario one). In Part II, the number of proficiency levels 
was reduced from five to three to provide more concrete descriptors of behaviors for the 
rating scale. To reduce ambiguity between proficiency levels, explicit definitions and 
descriptions needed to be placed below each proficiency level. In Part III, each BARS 
behavior factor was changed to focus on one indicator that would elicit a behavioral 
response. For example in Figure 18, the lED emplacement indicator in Part III is an ant 
trail. The associated response by the Marine squad leader to the lED emplacement 
indicator provided an improved characterization of the Marine squad leader’s perception 
and decision-making skill. SME interviews elicited observable behavior factors that 
indicated the decision-making proficiency of expert, competent, and novice decision¬ 
makers. 
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Cognitive Task: Notice Indicators of an lED Device 


Decision: Assess Primary lED Emplacement Indicator (5) 3B 
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NOTES : 


Figure 18. BARS Version 2 - Assess Primary lED Emplacement Indicator @ 3B. 


Mission: Link up with ANSF 
patrol IVO of the market 
lOT assist them in securing 
the area around an lED 
that they have identified. 

Atmospherics : ANSF 
having trouble keeping 
locals clear of lED. 

Visuais : lED covered with 
disturbed earth in the 
market. 

-Pressure plates IVO lA 
and 3B lEDs. 

Enemy Activity: 

-lA and/or 3B IE Ds 
detonate if pressure plates 
are stepped on. 


3/7 TD-1, Indoor Scenario 1: QRF To ANSF lED Find (Sqd Level) 
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Figure 19. QRF To ANSF lED Find with Final IIT Data Collection BARS Assessment 

Locations (After Fennell, 2013B). 
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5. Subject Matter Expert Interviews 

SME interviews were crucial to the development of the IIT decision-making 
performance metrics. The purpose of the interviews was to elicit indicators of decision¬ 
making behavior linked to BARS stratification levels (novice, competent, and expert). 
The format of the interview script afforded a conversational approach with the SMEs. 
The interview’s two-way dialog allowed the SMEs to be actively engaged in both 
direction and depth of the topics presented 

Each interview session was expected to take approximately two hours to 
complete. Development of the metrics required multiple salient perspectives to define 
the descriptors included in the BARS. Analysis of the interviews provided the necessary 
descriptors of an expert (high proficiency), competent (standard proficiency), and novice 
(rudimentary proficiency) decision-makers. 

a. Participants 

Developing behavioral descriptors for the decisions included in BARS 
required relevant subject-matter expertise in the task domain. Collecting this information 
via structured personnel interviews was used to elicit the SMEs knowledge about 
observable behaviors related to an lED-defeat task. 

Participants in this research were to be active-duty NCOs, staff non¬ 
commissioned officers (SNCOs), or commissioned officers with relevant ground combat 
arms experience. Uniformed service members with backgrounds or billets in quick 
reaction forces (QRE), special operating forces (SOE), or as infantry instructors 
(Coyotes) were also included. Additionally, SMEs must have deployed to an 
expeditionary environment where they were entitled to imminent danger pay within the 
last forty-eight months. 

b. Equipment 

Interviews were audio recorded at the approval of the participants to assist 
the investigators with their data collection and analysis. All information pertaining to 
personally identifiable information was secured in a manner consistent with the IRB 
protocol. 
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6 . 


Post-subject Matter Expert Interviews Behaviorally-Anchored Rating 
Scales Development 


Six SME interviews were conducted over a two-week period. These interviews 
provided the attributes necessary to discriminate between each level i.e., novice, 
competent, and expert decision-making proficiency. Participants were provided the 
training scenario (see Figure 6), and their roles and responsibilities for the interview were 
explained. In Part III of Figure 20, observable behaviors were synthesized from the SME 
interview data collection. The number of BARS was reduced from thirty-nine to six 
post-SME interviews BARS. These six BARS were used in the IIT final data collection 
phase. 
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Decision: Assess Indicators of Danger Areas Between C1/C4 


Novice 

- Ineffective applkstion 

-f'io Change in Formation 
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Post Security 
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Tactical Movement 


Competent 
• Effective Application 

Danger Area Inciicator : 

Chokepoint 

-Hesitates Change 
Formation 

-Lacks Dispersion 

-Constant Tactical 
Movement speed 


Eicpeit 

- Highly effective application 

-Fluid Formation Change 
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Satellite 

-AdjustsTactical Movement 
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Decision: Assess Indicators of Danger Areas Between A6/A7 


Novice 

Ineffective application 


-No Change in Formation 

-Doesn't Indiv/FT Bound & 
Post Security 

-Walks or Strolls through 


1 

Competent 
- Effective Application 

Danger Area Inciicator: 

Chokepoint 

-Hesitates Change 
Formation 

-Lacks Dispersion 

-No Adjustment in Pace 


Expert 

- Highly effective application 


-Fluid Formation Change 

-Proper Indiv/FT Bound & 
Satellite 

-Slow MovementThrough DA 


Figure 20. BARS Version 3 - Assess Indicator of Danger Area between C1/C4 and 

A6/A7. 
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B. IIT FINAL DATA COLLECTION PHASE 


The IIT final data collection occurred during the first two of the four-day training 
program. All observations and assessments focused on training-day one, indoor (Phase I) 
scenario one training storyboard (see Figure 19). During this scenario, six behaviorally- 
anchored rating scales (BARS) were linked to six observational indoor (Phase I) 
locations. The following measures section elaborates on the specific BARS used for data 
collection in this scenario. The metrics were printed and laminated on 8.5” x 11” paper. 
All observations and assessments were conducted in either the MILESTONE operator’s 
room (located in the IIT administration building) or the Phase I AAR Room. Prior 
coordination with the IIT Operations Officer assured that the same name scenario run 
would be utilized by all of the training participant’s squad elements. Observations and 
assessments were conducted in a manner to minimize participants’ awareness of the 
researcher. The anticipated number of scenario runs to be observed ranged from four to 
eight. 


1. Participants 

The research participants in this phase are from, 3'^'^ Marines, 7* Regiment based 
out of Marine Corps Base 29 Palms. There are approximately 120 to 180 Marines 
including support personnel in a Marine infantry company. The company has three 
infantry line platoons (nine squads) and a supporting weapons platoon. Observations 
were focused on the nine to twelve Marine squad leaders of the company. 

2. Equipment 

The observations and ratings occurred in one of two locations, the Phase I AAR 
Room or the MIEESTONE Operator’s room. MIEESTONE system is an improved 
instrumented video-capturing system replacing the existing Video Elashlight/TVCS 
systems used in the Phase I and II respectively. Currently, the Phase I facility is the only 
training range in the Marine Corps that uses the MIEESTONE system. The MIEESTONE 
Operator’s room is located in the IIT administration building (see Eigure 15). 
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3. 


Measures 


Six BARS were utilized in this phase. Figure 19 depicted the approximate 
location where the observations were focused occurred. The BARS results and 
discussion are described in the following results chapter of this thesis. 


73 



THIS PAGE INTENTIONALLY LEET BLANK 


74 



V. RESULTS 


This chapter presents results from the infantry immersion trainer (IIT) final data 
collection phase. 

A. MEASURES 

The first four BARS mentioned below were considered location-specific 
and are depicted in white text boxes in the Figure 19. Additionally, the final two BARS 
mentioned below were considered event specific and are represented by yellow text 
boxes in the storyboard. Loeation-specific BARS were pertinent only to specific areas or 
locations in the scenario, whereas event-specific BARS were only applicable when 
simulated improvised explosive devices (lEDs) were detonated in the scenario. 

As the squad navigated from the southern entrance of the indoor (Phase I) facility 
they encountered a chokepoint danger area between the Cl and C4 buildings. This 
chokepoint was the first major decision point encountered by the Marine squad leader. 
The chokepoint forced the Marine squad leader to recognize and tactically respond to the 
characteristics presented in the danger area, such as the channelizing features of the L- 
shaped alleyway, blind corners where visibility is obstructed, and the perpendicular 
alleyway between Cl and C3 buildings (see Figure 19). 

The Marine squad leader used his communication, location and proximity, and 
techniques, tactics and procedure (TTP) execution to tactically maneuver the squad 
through the C1/C4 chokepoint. Novice, competent, or expert proficiency ratings were 
attributed to the associated level of proficiency descriptors listed in Part I (red bold font) 
of Figure 21. As the squad encountered this first designated rating location, the rater 
used the descriptors under the rating scale in Part III (red bold font) to determine the 
Marine squad leader’s proficiency level regarding the following decision-making 
behaviors: ehange in formation, dispersion through the danger area, and tactical 
movement speed. The rater then associated the decision making behavior with the 
decision making proficiency level reflected by the behavior. 
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Decision: Assess Indicators of Danger Areas Between C1/C4 



0 
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1 

2 

Novice 
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Expert 

- Ineffective application 

- Effective Application 

- Highly effective application 


Danger Area Indicator : 

Chokepoint 


-No Change in Formation 

-Doesn't Indiv/FT Bound & 
Post Security 

-Demonstrates Lack of 
Tactical Movement 


-Hesitates Change 
Formation 


-Lacks 




-Constant Tactical 
Movement speed 


-Fluid Formation Change 

-Proper Indiv/FT Bound & 
Satellite 

-Adjusts Tactical Movement 
Speed effectively 


Figure 21. Final BARS: Assess Indicators of Danger Areas Between C1/C4 - 

Chokepoint. 


Once the decision making behaviors were assessed, they were rated, marked and 
then averaged. When a Marine squad leader had a majority of ratings (Part III in Figure 
21) linked to a particular proficiency level, i.e., 0, 1, or 2 (Part I of Figure 21), that 
proficiency category is awarded (novice, competent, expert). If there were equal 
distributions of observed behaviors for the proficiency levels, the lesser rating will be 
awarded. As shown in Figure 22, there was one novice and one competent rating given, 
and the third behavior was inconclusive or not observable. Since the cumulative average 
((O-hl) / 2 = .5) were less than competent rating (<1), a novice rating were awarded. 
However, if three proficiency descriptors were represented equally across the novice, 
competent and expert categories, the competent rating would be awarded. For example, 
in Figure 23, there was one novice, one competent, and one expert rating indicated by the 
red circles. The average scoring of the three ratings ((0 -i- 1 -i- 2) / 3 = 1) means that the 
Marine squad leader was rated a competent decision-making proficiency category for that 
task. 
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Decision: Assess Indicators of Danger Areas Between C1/C4 
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Figure 22. Final BARS: Assessment seoring of the Indieators of Danger Areas between 

C1/C4 - Chokepoint. 


Decision: Assess Indicators of Danger Areas Between C1/C4 
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Figure 23. Final BARS: Assessment scoring of the Indicators of Danger Areas between 

C1/C4-Chokepoint. 
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As the squad navigated from the C1/C4 chokepoint toward the bazaar, they likely 
maneuvered through the chokepoint between the A6 and A7 buildings (See Figure 19). 
The Marine squad leader either navigated the squad through the A6/A7chokepoint or 
decided to use an alternate path. Instead of navigating through the A6/A7 chokepoint, the 
Marine squad leader may instead satellite his individual fire teams. Satelliting is a 
tactical procedure where the Marine squad leader breaks up the squad into autonomous 
fire team elements. The BARS shown in Figure 5 assumes that the Marine squad leader 
and some portion of the squad navigated through the A6/A7 chokepoint. 

As shown in Figure 24, the A6/A7 and C1/C4 chokepoints have identical 
descriptors and were rated in a similar manner. The differences between the two 
chokepoints were their physical characteristics and proximity to the bazaar objective area. 
The C1/C4 chokepoint was a more straight-forward danger area compared to the complex 
characteristics of the A6/A7 chokepoint (“L” shaped vs. “T” shaped chokepoints). A 
danger area such as the C1/C4 chokepoint allows the Marines to break-up the location 
into sequential parts. Conversely in the A6/A7 chokepoint, the Marines have to contend 
with two potential avenues of approach represented by the top of the “T” shape. Finally, 
the proximity of the bazaar to the A6/A7 chokepoint creates a natural draw and friction 
point for the Marine squad leader and his squad. 
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Decision: Assess Indicators of Danger Areas Between A6/A7 
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Figure 24. Final BARS: Assess Indicators of Danger Areas between A6/A7 - 

Chokepoint. 


The squad’s mission in this scenario was to tactically navigate to the 
bazaar, link-up with the Afghan national security forces (ANSF) located there, identify 
the point of origin of any suspected lED(s), establish a cordon and security around the 
device(s), and coordinate with explosive ordinance disposal (EOD) to disarm the lED(s). 
The BARS focused on the Marine squad leader’s decision making in controlling and 
securing the internal and outer area of the cordon. As depicted in Eigure 25, the objective 
of the BARS was to rate the Marine squad leader’s observable decisions related to 
securing avenues of approach (AOAs), the lED marking plan, fire team security 
execution, and leveraging the ANSE. 
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Decision: Assess Indicators of Suitable Cordon 
Environment 
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Figure 25. Final BARS: Assess Indicators of Suitable Cordon Environment: Box Cordon 

and Security Surrounding Detected lED. 


As depicted in Figure 26, the focus of the BARS was the Marine squad leader’s 
decision making ability to coordinate with EOD to disarm the cordon’s simulated lED(s). 
The BARS decision-making descriptors included the EOD link-up plan, lED marking 
plan, civilian’s location and proximity to the cordon, and employment of the squad’s 
mine sweeping equipment. 
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Decision: Assess Indicators of Suitable Cordon Environment 
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Figure 26. Final BARS: Assess Indieators of Suitable Cordon Environment: 

Coordination with EOD until lED disarmed. 


The following two BARS were event speeifie, rather than location specific in the 
scenario. Event-specific BARS were concerned with the simulated unexploded 
ordinance (UXO), related to the scenario lEDs and mines. Detonations of the simulated 
UXOs were at the prerogative of the accompanying trainer. With that responsibility, the 
trainer determined the simulated UXOs effective casualty range and its blast-related 
effects. When a detonation occurred, the Marine squad leader had two immediate 
decisions. Eirst, he must secure the local area in order to mitigate and prevent subsequent 
engagements. These UXO detonations have operated in combination with other types of 
attacks such as small arms fire, or secondary lEDs and mines. These combination attacks 
are intended to create as many casualties as possible. 

As depicted in Eigure 27, the Marine squad leader’s second decision related to the 
inquiry of the squad’s medical status, timeliness of medical evacuation/ casualty 
evacuation (MEDIVAC/CASEVAC) 9-Eine report, MEDIV AC/CASE VAC reporting 
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fluency, and radio conduct relating to tone, cadence, clarity, and brevity. The UXO/IED 
9-Line report to higher headquarters must always be completed after an UXO detonation. 


Decision: Assess Indicators of Reactive Post-Detonation 



0 

1 

2 

Novice 

Competent 

Expert 

- Ineffective application 

- Effective Application 

- Highly effective application 


Reactive Post-Detonation Indicator : 

Personnel Medical Status & Reporting Considerations 


-Doesn't Give CASEVAC 
Report 

-Delayed I nquiring Medical 
Status 

-Report doesn't match 
format 

-Vague, Erratic, Unclear 
Communication 


-CASEVAC Report Delayed 

-Medical Status hesitant 

-Report Given w/ 
Reference 

-Adequate Communication 
is Rushed 


-CASEVAC Report is Timely 

-Immediately Request 
Medical Status 

-CASEVAC Report is 
Memorized 

-Calm, Cool, Collected 
Communication 


Figure 27. Final BARS: Assess Indicators of Reactive Post-Detonation: Personnel 

Medical Status & Reporting Considerations. 


After simulated UXO detonations. Marine squad leaders were expected to 
send two radio reports higher headquarters. Their first priority was the 
CASEVAC/MEDIVAC 9-Line report (assuming there were casualties), followed by the 
lED/UXO 9-Line report. Both standardized reports were to be sent independent of one 
another. As depicted in Figure 28, this BARS was used to rate the four behavioral 
descriptors of the Marine squad leader. The behavioral descriptors related to the 
timeliness of the lED/UXO 9-Line report, the completeness of the lED/UXO 9-Line 
report (including remarks), lED/UXO 9-Line reporting fluency, and radio conduct with 
respect to tone, cadence, clarity, and brevity. 
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Decision: Assess Indicators of Reactive Post-Detonation 
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Figure 28. Final BARS: Improvised Explosive Device Classification and Reporting 

Considerations. 


B. RESULTS 

Table 6 presents the results of evaluations of four Marine squad leaders during 
training day one and two. A total of six scenario run observations were made during the 
week. However, two of the observations were excluded from analysis due to major 
modifications that were made to the scenario. These modifications included an ad hoc 
inclusion of small-arms fire (SAF) engagements, changing the objective area and lED 
locations. As depicted in Table 6, the remaining four assessments are listed along the top 
of the table. The BARS titles are listed along the left-hand side of the table. The 
assessment score for each Marine squad leader was indicated by a numerical “0” 
(novice), “1” (competent), or “2” (expert). 
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Table 6. Number of BARS Assessments. 



Unit 

BARS: Indicator 

1*‘ Sqd 

2”“ Sqd 

1*‘ Sqd 

2"“ Sqd 

Assess indicators of danger areas between 
C1/C4: Chokepoint 

0 

2 

2 

2 

Assess indicators of danger areas between 
A6/A7: Cbokepoint 

1 

N/A 

2 

N/A 

Assess indicators of suitable cordon 
environment: Box cordon and security 
surrounding detected lED 

0 

0 

2 

0 

Assess indicators of suitable cordon 
environment: Coordination with EOD until 
lED disarmed 

1 

0 

2 

1 

Assess indicators of reactive post¬ 
detonation: Personnel medical status & 
reporting considerations 

No lED 
Detonation 

2 

No lED 
Detonation 

0 

Assess indicators of reactive post¬ 
detonation: lED classification and reporting 
considerations 

No lED 
Detonation 

2 

No lED 
Detonation 

0 


An additional post hoc assessment of each squads BARS was conducted by the 
individual IIT trainer who accompanied each squad. Each of the four scenario runs was 
accompanied by a different trainer; this provided an independent evaluation of each 
squad leader’s BARS scores. Each trainer was asked to compare their evaluation of the 
squad leader’s performance with the BARS assessment and then asked to confirm or 
disagree with the results indicated by each BARS. Results indicate that the IIT trainers 
generally agreed with the individual BARS or did not contradict the individual metric’s 
as presented. 

C. DISCUSSION 

The goal of this thesis was to develop, refine and employ infantry domain relevant 
BARS in order to examine their utility for evaluating Marine squad leaders’ decision¬ 
making performance. During the final data collection phase, six BARS were used for 
rating the decision making performance of four Marine squad leaders. All observations 
were based on one training scenario. Only two of the four scenario training runs involved 
a reactive detonation of a simulated lED. This meant that the “Assess indicators of 
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reactive post-detonation: Personnel medical status & reporting considerations” and 
“Assess indicators of reactive post-detonation: lED classification and reporting 
considerations” BARS were not used for scenario runs that did not have a reactive 
detonation. Additionally, in two of the scenario runs, the Marine squad leader elected not 
to navigate through the A6/A7 chokepoint. If it was observed that the Marine squad 
leader did not personally survey or encounter the chokepoint, the “Assess indicators of 
danger areas between A6/A7: Chokepoint” metric was not employed. 

The limitations of the analysis presented above not notwithstanding, results 
support the utility of BARS as a practical method for the assessment of Marine squad 
leaders’ decision-making performance. The findings presented here are particularly 
encouraging in that they demonstrate the realistic possibility of replacing the nonstandard 
and subjective methods currently employed in LVC infantry training environments such 
as the IIT. By providing instructors and evaluators the ability to use standardized metrics 
such as BARS for small-unit decision-making training scenarios, the opportunity for 
standardizing after action reviews can improve trainee feedback and remediation. 

BARS assessment results were corroborated by the IIT trainer who accompanied 
each squad. Each of the four scenario runs was accompanied by a different trainer. Each 
trainer was independently shown the Marine squad leader’s assessment scores (six) and 
was asked to confirm or disagree with the metrics’ findings. All the IIT trainers generally 
agreed with or did not contradict the individual metric’s findings. 

1. Lessons Learned from Thesis Research 

There were several lessons learned from this thesis research and are discussed in 
the following: 

• There exists little consensus in the BARS literature on the metric’s format and 
structure. Additionally, there was a lack of empirical applications of BARS. 
Eurther, reference in the literature to assist in developing BARS for applied 
domains is sparse. 

• An emphasis on prior coordination is essential when conducting field research 
at an operational training range. Eimited knowledge of IIT scenarios and 
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operations hampered assessments and ratings during the second IIT visit. 
During the third visit to the IIT, limited awareness of the scenario run 
scheduling and order inhibited set up and preparation for each event. The 
researcher had to constantly request status updates and confirmations from the 
IIT staff regarding scenario run times and squad identification. 

• Ad hoc changes by the IIT staff during the third visit, prevented inclusion of 
two of the six scenario run observations. There was little warning of the 
changes prior to their introduction into the scenario run. Since training 
participant leadership drove the training objectives, the IIT staff made 
modifications on the fly to reflect their intent or wishes. The two excluded 
scenario runs included modifications early into the scenario runs. The 
changes adversely affected the researcher’s ability rate the Marine squad 
leader and made using the BARS irrelevant. 

• During the BARS development, there was a consistent trial-and-error theme to 
the BARS. During the second visit to the IIT, it was discovered that some 
BARS that were supposed to be independent were actually nested into the 
behavioral descriptors of other BARS. For example, in the “Consider how to 
Respond Immediately to Detonation” BARS (Figure 14), the fourth behavior 
factor focused on assessing the establishment of a cordon around an lED. As 
previously mentioned, the “Determine How to Establish Security and 
Cordons” BARS (Eigure 13) is an independent BARS and cognitive task. 

• Due to the restrictions incorporated in the institutional review board (IRB) 
research protocol, personal observations of the squads engaged in training was 
not permitted for data collection. All observations and ratings occurred while 
watching video monitors of the squads engaged in training. Having the prior 
agreements and coordination with the training participants and the NFS IRB 
would have greatly improved the quality of the results. 
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VI. RECOMMENDATIONS AND EUTURE WORK 


This thesis research demonstrated a deliberate approach toward developing 
behaviorally-anchored rating scale (BARS) assessments. Behavioral responses assessed 
were specific to Marine small-unit decision making. The introduction, background, and 
literature review chapters described the motivation and scope of the metric development 
process. The methodology and results chapters encompassed the empirical application of 
the method. The following sections elaborate on the BARS development and future work 
recommendations. 

A. BEHAVIORALLY-ANCHORED RATING SCALES DEVELOPMENT 

RECOMMENDATIONS 

This section presents recommendations from the BARS development process. 
Several impediments were encountered during the BARS development process which 
impacted bringing this assessment method to reality. The following recommendations 
are based on the researcher’s experience during the BARS development process. 

• Research is more than finding the answer - The goal of research should not 
be exclusively focused on the end state of the results. More often, scientific 
insight is garnered from application of the scientific process that can 
empirically fail or go awry. The scientific method is predicated on the 
replicative process, rather than the process’s product. One goal of this 
research was to ensure the process is repeatable in order to sustain research 
findings. In other words, the answer is less important than the information 
discovered from the analysis. In this BARS development process, there was 
an initial preoccupation with trying to make the BARS relevant to all training 
scenarios. The more appropriate approach was to make a single metric 
applicable to a specific scenario and determine its relevance. Occam’s razor 
exemplifies this concept with “all other things being equal, the simpler 
explanation is preferable” (Blumer, Ehrenfeucht, Haussler, & Warmuth, 1987, 
p. 377). Scoping the BARS to a single training scenario was a more effective 
approach to rate decision-making for this thesis. 

• Making lemonade when the world gives you lemons - Thesis research was 
not dissimilar to combat, in that “no plan of operations extends with certainty 
beyond the first contact with the main hostile force” (Vego, 2007, p. IX-63). 
Reviewing research literature and incorporating guidance from thesis advisors 
assisted the thesis acclimatization process to a point. Beyond planning, this 
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thesis research required a creative and adaptable plan capable of contending 
with several expected research difficulties. 

• During the BARS development process, issues arose during the analysis of 
subject-matter expert (SME) interview data. These issues were related to 
contradictions in SME responses such as, when SMEs independently refuted 
other SME assertions. Differences in data collected to develop decision¬ 
making attributes produced indecision about which SME responses were more 
valid or accurate. Compounding this dilemma were additional constraints 
related to limited personal or professional experience in the domain (that is, 
the researcher was a Marine aviator, not a Marine infantry officer). Without 
an objective reference, literature became the sole arbiter to the inconclusive 
data. The takeaway from this was that opinions, even informed opinions, are 
nothing more than experience-based heuristics. However, justifying 
discrepancies between opinions gathered from SME interviews proved to 
quite challenging and resulted in unanticipated delays in the thesis progress. 

• Choose your path carefully - The NFS student’s biggest educational career 
decision is the selection of a thesis topic. Selecting a thesis topic is usually 
based on personal experiences that motivate the researcher’s interest. The 
overarching intent of this research was to enhance small-unit training and 
assessment methods to facilitate protecting military service members. The 
goal for this thesis was to objectively evaluate small-unit decision making 
performance in the lED-defeat domain. 

Selection of a ground combat-arms related thesis came from experiences 
relating to deployments to Iraq and Afghanistan. After witnessing some of 
the carnage and loss of life resulting from improvised explosive devices 
(lEDs), the topic was a means to confront my personal war experiences. 
Technological advances have improved the means to neutralize and defeat 
lEDs, but human ability lags considerably behind. Perception, 
interpretation, and recognition of lED emplacement characteristics are an 
enduring Marine training thrust. While the research goal was not meant to 
eliminate the evaluator’s perspective, the focus was to overlay objective 
criteria on the trainer’s assessment in order to structure the after-action 
review of training conducted. 

B. FUTURE WORK 

During the creative process, contingencies arise which are never fully explored 
for a myriad of reasons. These discoveries are usually shelved for future work by 
subsequent researchers. Eor example, with the limited availability of SMEs, there were 
difficulties in substantiating the decision-making behaviors specific to the IIT training 
participants. Euture research should develop and refine the behaviors under scrutiny 

through a controlled study or experiment. Having the specific behaviors, such as those 
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identified here, significantly correlated to training participants’ performance will not only 
enhance the quality of the BARS development for the small unit decision making 
training. As shown in Table 7, the recommendations explore different avenues where the 
existing thesis research could be advanced. 


Table 7. Follow-on Research Recommendations. 


RECOMMENDATIONS 

DESCRIPTIONS 

1. Behaviorally-Anchored 
Rating Scales 
Experimentation 

This thesis research developed an empirical concept 
without statistically verifying the reliability or validity of 
the BARS. The next phase of this research would be to 
design a study to evaluate this BARS assessment 
method. The purpose of the experiment would be to 
determine the level of confidence in this assessment 
method compared to other approaches. Additionally, a 
study could involve multiple individuals as assessors to 
evaluate the reliability of the results. 

2. Behaviorally-Anchored 
Rating Scales 

Development in Alternate 
Infantry Immersion 

Trainer locations 

This BARS development process was only relevant to 
the Marine Corps Base Camp Pendleton (MCBCP) 
infantry immersion trainer (IIT). With two additional 
facilities at Camp Lejeune, NC, and Kaneohe Bay, HI, 
additional research could validate the BARS approach at 
the other IIT facilities. Additional experimentation could 
also assess the level of training effectiveness at each 
training range. 

3. Validation of Behaviorally 
Indicators for Small-Unit 
Decision Making 

Behavioral indicators of decision making were 
synthesized from six participants in the SME interviews. 
Complementary research could additionally determine 
the full-spectrum of behavioral indicators for a 
statistically representable population. The initial intent 
would be to make the indicators valid only to IIT 
facilities and then expand the research’s scope to other 
training ranges. 


This thesis research has two logical extensions to the current research 


methodology for the near-term (0-15 years), and long-term (>15 years) regimes. As 
shown in Table 8, the following are the natural projections of the research in those 
categorical timeframes. 
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Table 8. Near- and Long-Term Future Work Recommendations. 


RECOMMENDATIONS 

DESCRIPTIONS 

1. (Near-Term) Behaviorally- 
Anchored Rating Scales 
Assessment Expansion 

The near-term recommendation of this BARS 
assessment methodology is to have the IIT trainers 
integrate BARS as part into their assessment process. 
Technological advances and improvements to the BARS 
would enable the IIT staff to iterate the BARS as 
scenarios evolved over time. Additionally, the 

incorporation of granular learning objectives to the 
scenario training run (establishing and securing cordon 
around a detected lED) would enhance the critique. 

2. (Long-Term) 

Decentralized 

Behaviorally-Anchored 
Rating Scales Assessment 
System 

The long-term recommendation for the BARS 
assessment methodology would be a technological 
incorporation that permits military organizations the 
ability to perform organic assessments of small-unit 
decision making. This would enable Platoon 

Commanders to use BARS to assess their platoon’s 
decision making (and other) training without the need 
for external support. 
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